Apparatus, system, and method for a front-end, distributed raid

ABSTRACT

An apparatus, system, and method are disclosed for a front-end, distributed redundant array of independent drives (“RAID”). A storage request receiver module receives a storage request to store object or file data in a set of autonomous storage devices forming a RAID group. The storage devices independently receive storage requests from a client over a network, and one or more of the storage devices are designated as parity-mirror storage devices for a stripe. The striping association module calculates a stripe pattern for the data. Each stripe includes N data segments, each associated with N storage devices. The parity-mirror association module associates a set of the N data segments with one or more parity-mirror storage devices. The storage request transmitter module transmits storage requests to each storage device. Each storage request is sufficient to store onto the storage device the associated data segments. The storage requests are substantially free of data.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/204,436, entitled “Apparatus, System, and Method for a Front-EndDistributed RAID,” filed on Sep. 5, 2011, which is a continuation ofU.S. patent application Ser. No. 11/952,116, entitled “Apparatus,System, and Method for a Front-End Distributed RAID,” filed on Dec. 6,2007, and issued as U.S. Pat. No. 8,019,940, and which claims priorityto U.S. Provisional Patent Application No. 60/873,111 entitled“Elemental Blade System” and filed on Dec. 6, 2006, for David Flynn, etal., and U.S. Provisional Patent Application No. 60/974,470 entitled“Apparatus, System, and Method for Object-Oriented Solid-State Storage”and filed on Sep. 22, 2007, for David Flynn, et al., each of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to data storage and more particularly relates todata storage using a front-end, distributed redundant array ofindependent drives (“RAID”).

2. Description of the Related Art

Traditional RAID systems are configured with a RAID controller thatfunctions to receive data, calculate striping patterns for the data,divide the data into data segments, calculate a parity stripe, store thedata on storage devices, update the data segments, etc. While some RAIDcontrollers allow some functions to be distributed, the storage devicesmanaged by the RAID controller do not communicate with clients directlyfor storing data striped in a RAID. Instead storage requests and datafor RAIDing pass through the storage controller.

Requiring the RAID controller to touch all of the data to be stored in aRAID is inefficient because it creates a dataflow bottleneck. This isespecially true during a read-modify-write process where bandwidth andperformance of all of the drives in the RAID group is consumed whileonly a subset is actually updated. In addition, a region of the storagedevice designated for data managed by the RAID controller is typicallydedicated to the RAID group and cannot be accessed independently. Accessto a storage device by a client must typically be accomplished bypartitioning the storage device. Where partitioning is used, partitionsaccessible for general storage are not used for RAID and partitionsallocated to the RAID group are not accessible for general data storage.Schemes that oversubscribe partitions in order to globally optimizeutilization are complex and more difficult to manage. In addition,storage space allocated for one RAID group cannot be accessed by morethan one RAID controller unless one is designated as master and otherRAID controllers act as slaves unless the master RAID controller isinactive, non-functional, etc.

Typical RAID controllers also generate parity data segments outside ofthe storage devices of the RAID group. This can be inefficient becauseparity data segments are typically generated and then are sent to astorage device for storage, which requires computing capacity of theRAID controller. Tracking parity data segment location and updates mustalso be done at the RAID controller instead of autonomously at a storagedevice.

Where it is necessary to ensure that the data remains available if theseparate RAID controller is offline, RAID controllers are typicallycross connected to the drives and to each other, and/or mirrored ascomplete sets, making data availability expensive and difficult tomanage, and dramatically reducing the reliability of the storagesubsystem.

SUMMARY OF THE INVENTION

What is needed is a system, apparatus, and method for front-enddistributed RAID that allows RAIDing on a per data segment, per object,per file, or similar basis and that eliminates the need for RAIDcontrollers and RAID controller couplets situated between the client andthe storage devices. In such a system, apparatus, and method, a RAIDgroup can be created for one data segment, object, or file and managedwithin one group of storage devices by one RAID controller while asecond RAID group may be created for another data segment, object, orfile that encompasses some of the same storage devices of the first RAIDgroup. The RAID control functions may be distributed among clients, athird-party RAID management device, or among storage devices. Thefront-end distributed RAID system, apparatus, and method may also sendcommands to storage devices of a RAID group and may allow the storagedevices to directly access and copy data through direct memory access(“DMA”), or remote DMA (“RDMA”).

The present invention has been developed in response to the presentstate of the art, and in particular, in response to the problems andneeds in the art that have not yet been fully solved by currentlyavailable RAIDs. Accordingly, the present invention has been developedto provide an apparatus, system, and method for high, performance, RAIDSthat overcome many or all of the above-discussed shortcomings in theart.

The apparatus is provided with a plurality of modules including astorage request receiver module, a striping association module, aparity-mirror association module, and a storage request transmittermodule. The storage request receiver module receives a storage requestto store data in a storage device set. The data is from a file or anobject. The storage device set includes autonomous storage devicesforming a RAID group. The storage devices independently receive storagerequests from a client over a network. One or more of the autonomousstorage devices within the storage device set are designated asparity-mirror storage devices for a stripe.

The striping association module calculates a stripe pattern for thedata, where the stripe pattern includes one or more stripes. Each stripeincludes a set of N data segments and associates each of the N datasegments with one of N storage devices in the storage device setassigned to the stripe. The parity-mirror association module associatesa set of the N data segments with one or more parity-mirror storagedevices in the storage device set. The one or more parity-mirror storagedevices are in addition to the N storage devices. The storage requesttransmitter module transmits one or more storage requests to eachstorage device in the storage device set. Each storage request issufficient to store onto the storage device the one or more datasegments associated with the storage device receiving the storagerequest. The storage requests are substantially free of data identifiedby the storage requests.

In one embodiment, the parity-mirror association module associates a setof N data segments with each parity-mirror storage device. In a furtherembodiment, the apparatus includes a front-end parity generation modulethat calculates, independent of a client, a parity data segment for thestripe and stores the parity data segment on a parity-mirror storagedevice. The parity data segment is calculated from the set of N datasegments on the parity-mirror storage device. In yet a furtherembodiment, the front-end parity generation module calculates the paritydata segment within at least one of one or more storage devices of thestorage device set and a third party RAID management device. In anadditional embodiment, the front-end parity generation module operateswithin a parity-mirror storage device. In another embodiment, thefront-end parity generation module calculates the parity data segmentand transmits the calculated parity data segment to one or moreadditional parity-mirror storage devices in a second set of storagedevices forming a mirror.

In one embodiment, the apparatus includes a data segment recovery modulethat recovers a data segment stored on a storage device of the storagedevice set in response to a request to read the data segment, or thedata comprising the data segment, where the storage device isunavailable. The data segment may be recovered using the data segmentson available storage devices of the storage device set. The data segmentmay be recovered using a combination of the parity data segments and thedata segments that are on available storage devices of the storagedevice set. The data segment may be recovered using a mirror storagedevice that contains a copy of the data segment. The mirror storagedevice includes one of a set of one or more storage devices storing acopy of the N data segments.

In a further embodiment, the data segment recovery module operates andrecovers a data segment while residing on the client, a third party RAIDmanagement device, or one or more storage devices of the storage deviceset. A mirror storage device contains a copy of the data segment and themirror storage device includes one of a set of one or more storagedevices storing a copy of the N data segments.

In one embodiment of the apparatus, the apparatus includes a datarebuild module that stores a recovered data segment on a replacementstorage device in a rebuild operation. The recovered data segmentmatches an unavailable data segment stored on an unavailable storagedevice of the storage device set. The rebuild operation is to restoreone or more of data segments and parity data segments onto thereplacement storage device to match data segments and parity datasegments stored previously on the unavailable storage device.

The recovered data segment may be recovered for the rebuild operationusing the available data segments on available storage devices of thestorage device set. The recovered data segment may be recovered using acombination of a parity data segment from one or more of theparity-mirror storage devices and the available data segments onavailable storage devices of the storage device set. The recovered datasegment may be recovered using a matching data segment that matches theunavailable data segment, where the matching data segment is read from aparity-mirror storage device.

The recovered data segment may be recovered using a matching datasegment that matches the unavailable data segment, where the matchingdata segment is read from a mirror storage device containing a copy ofthe data segment. The mirror storage device includes one of a mirroreddevice set of one or more storage devices storing a copy of the N datasegments. In a further embodiment, the data rebuild module operates andstores a recovered data segment while residing on a client, a thirdparty RAID management device, one or more storage devices of the storagedevice set, or a mirrored device set.

The apparatus is further configured, in one embodiment, to include aparity rebuild module that rebuilds the recovered parity data segment ona replacement storage device in a rebuild operation. The recoveredparity data segment matches an unavailable parity data segment stored onan unavailable parity-mirror storage device assigned to the stripe. Therebuild operation restores one or more of data segments and parity datasegments onto the replacement parity-mirror storage device to match datasegments and parity data segments stored previously on the unavailableparity-mirror storage device.

The parity data segment may be recovered by copying the parity datasegment stored on a parity-mirror storage device in a second set ofstorage devices storing a mirror copy of the stripe. The parity datasegment may be recovered by copying the parity data segment from aparity-mirror storage device in the storage device set. The parity datasegment may be recovered by generating the parity data segment using oneor more of the N data segments and parity data segments stored on theavailable storage devices of the storage device set and a mirror storagedevice containing a copy of the data segment. The mirror storage devicecomprising one of a mirrored device set of one or more storage devicesstoring a copy of the N data segments. In a further embodiment, theparity rebuild module operates and rebuilds the recovered parity datasegment while residing on the client, a third party RAID managementdevice, one or more storage devices of the storage device set, or amirrored device set.

In another embodiment of the apparatus, independently receiving storagerequests from a client for the autonomous storage devices of the storagedevice set includes receiving independent storage requests that requeststoring data into a region within the storage devices that is alsoavailable for storing data that includes data segments that are assignedby the striping association module, data segments that are assigned bythe parity-mirror association module, and/or parity data segmentscalculated by the front-end parity generation module, where data storedby executing at least one of the independent storage requests is notstored in a RAID configuration.

In one embodiment, the parity-mirror association module associates eachdata segment of a set of N data segments with a parity-mirror storagedevice, and the parity-mirror devices includes a mirror of the N datasegments on the N storage devices. In another embodiment, one or morestorage requests identify the storage devices that comprise the storagedevice set of the stripe. In yet another embodiment, the storage requestreceived by the storage request receiver module is substantially free ofthe data identified by the storage request. In yet another embodiment,each storage device manages data transfers, between the storage deviceand the client, of the one or more data segments and parity datasegments associated with the storage device using direct memory access(“DMA”) or remote direct memory access (“RDMA”).

In one embodiment of the apparatus, one or more data segments arebroadcast or multicast to two or more of the storage devices within thestorage device set and each storage device receiving the broadcast ormulticast stores the one or more data segments associated with thestorage device without storing data segments not associated with thestorage device. In an additional embodiment, the network comprises asystem bus.

In one embodiment, the striping association module calculates a stripepattern and associates each data segment of each stripe with one or moreof the storage devices of the storage device set while operating fromwithin at least the client, one or more storage devices of the storagedevice set, or a third party RAID management device. In a furtherembodiment, the parity-mirror association module associates a set of Ndata segments with one or more parity-mirror storage devices and thestorage request transmitter module transmits the storage requests whileoperating from within the client, one or more storage devices of thestorage device set, or a third party RAID management device.

In an additional embodiment, the apparatus includes a parity alternationmodule that alternates, for each stripe, which storage devices in thestorage device set are designated as the one or more parity-mirrorstorage devices for the stripe. In another embodiment, the apparatusincludes a peer-to-peer communication module that transmits and receivesstorage requests within the storage devices of the storage device setwith peer devices within the storage device set and outside the storagedevice set. In yet another embodiment, one or more of the autonomousstorage devices of the storage device set are allocated within a firstRAID group for at least a portion of a first object or file andallocated within a second RAID group for at least a portion of a secondobject or file. In an additional embodiment of the apparatus, thestorage request comprises an object request to store an object.

An alternate embodiment of the apparatus is provided with a storagerequest receiver module, a striping module, a parity-mirror module, astorage request transmitter module, and a front-end parity generationmodule. In this embodiment, the storage request receiver module receivesa storage request to store data in a storage device set. The data isfrom a file or an object and the storage request is substantially freeof data from the file or object. The storage device set includesautonomous storage devices forming a RAID group. The storage devicesindependently receive storage requests from a client over a network. Oneor more of the autonomous storage devices within the storage device setare designated as parity-mirror storage devices for a stripe.

The striping association module calculates a stripe pattern for thedata, where the stripe pattern includes one or more stripes and eachstripe includes a set of N data segments. The striping associationmodule associates each of the N data segments with one of N storagedevices in the storage device set assigned to the stripe. Theparity-mirror association module associates a set of the N data segmentswith one or more parity-mirror storage devices in the storage deviceset, where the one or more parity-mirror storage devices are in additionto the N storage devices.

The storage request transmitter module transmits one or more storagerequests to each storage device in the storage device set, where eachstorage request is sufficient to store onto the storage device the oneor more data segments associated with the storage device receiving thestorage request. The front-end parity generation module operates in aparity-mirror storage device that calculates, independent of a client, aparity data segment for the stripe and stores the parity data segment ona parity-mirror storage device. The parity stripe is calculated from theset of N data segments on the parity-mirror storage device.

In a further embodiment of the alternate apparatus, the storage requesttransmitter module transmits one or more storage requests to eachstorage device in the storage device set by broadcasting or multicastingthe storage requests to two or more of the storage devices within thestorage device set. Each storage device receiving the broadcast ormulticast executes one or more portions of the storage requestassociated with the storage device and ignores portions of the servicerequest not associated with the storage device. In another embodiment,one or more data segments are broadcast or multicast to two or more ofthe storage devices within the storage device set, and each storagedevice receiving the broadcast or multicast stores the one or more datasegments associated with the storage device without storing datasegments not associated with the storage device.

A system of the present invention is also presented. The systemsubstantially includes the modules and embodiments described above withregard to the apparatus. In particular, the system, in one embodiment,includes a storage request receiver module, a striping associationmodule, a parity-mirror association module, and a storage requesttransmitter module. The system also includes a storage device set ofautonomous storage devices forming a RAID group. The storage devicesindependently receive storage requests from a client over a network,where one or more of the autonomous storage devices within the storagedevice set are designated as parity-mirror storage devices for a stripe.

The storage request receiver module receives a storage request to storedata. The data is from a file or an object. The striping associationmodule calculates a stripe pattern for the data, where the stripepattern includes one or more stripes and each stripe includes of a setof N data segments. The striping association module associates each ofthe N data segments with one of N storage devices in the storage deviceset assigned to the stripe. The parity-mirror association moduleassociates a set of the N data segments with one or more parity-mirrorstorage devices in the storage device set, where the one or moreparity-mirror storage devices are in addition to the N storage devices.The storage request transmitter module transmits one or more storagerequests to each storage device in the storage device set. Each storagerequest is sufficient to store onto the storage device the one or moredata segments associated with the storage device receiving the storagerequest. The storage requests are substantially free of data identifiedby the storage requests.

The system, in one embodiment, may include a front-end parity generationmodule that calculates, independent of a client, a parity data segmentfor the stripe and stores the parity data segment on a parity-mirrorstorage device. The parity stripe is calculated from the set of N datasegments on the parity-mirror storage device.

A method of the present invention is also presented for highperformance, distributed RAID. The method in the disclosed embodimentssubstantially includes the steps necessary to carry out the functionspresented above with respect to the operation of the described apparatusand system. In one embodiment, the method includes receiving a storagerequest to store data in a storage device set, where the data is from afile or an object and the storage device set includes autonomous storagedevices forming a RAID group. The storage devices independently receivestorage requests from a client over a network, where one or more of theautonomous storage devices within the storage device set are designatedas parity-mirror storage devices for a stripe.

The method includes calculating a stripe pattern for the data, where thestripe pattern comprising one or more stripes and where each stripeincludes of a set of N data segments. The method includes associatingeach of the N data segments with one of N storage devices in the storagedevice set assigned to the stripe. The method includes associating a setof the N data segments with one or more parity-mirror storage devices inthe storage device set. The one or more parity-mirror storage devicesare in addition to the N storage devices. The method includestransmitting one or more storage requests to each storage device in thestorage device set, where each storage request is sufficient to storeonto the storage device the one or more data segments associated withthe storage device receiving the storage request. The storage requestsare substantially free of data identified by the storage requests.

In a further embodiment, the method includes calculating, independent ofa client, a parity data segment for the stripe and storing the paritydata segment on a parity-mirror storage device, where the parity stripeis calculated from the set of N data segments.

Reference throughout this specification to features, advantages, orsimilar language does not imply that all of the features and advantagesthat may be realized with the present invention should be or are in anysingle embodiment of the invention. Rather, language referring to thefeatures and advantages is understood to mean that a specific feature,advantage, or characteristic described in connection with an embodimentis included in at least one embodiment of the present invention. Thus,discussion of the features and advantages, and similar language,throughout this specification may, but do not necessarily, refer to thesame embodiment.

Furthermore, the described features, advantages, and characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. One skilled in the relevant art will recognize that theinvention may be practiced without one or more of the specific featuresor advantages of a particular embodiment. In other instances, additionalfeatures and advantages may be recognized in certain embodiments thatmay not be present in all embodiments of the invention.

These features and advantages of the present invention will become morefully apparent from the following description and appended claims, ormay be learned by the practice of the invention as set forthhereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the advantages of the invention will be readilyunderstood, a more particular description of the invention brieflydescribed above will be rendered by reference to specific embodimentsthat are illustrated in the appended drawings. Understanding that thesedrawings depict only typical embodiments of the invention and are nottherefore to be considered to be limiting of its scope, the inventionwill be described and explained with additional specificity and detailthrough the use of the accompanying drawings, in which:

FIG. 1A is a schematic block diagram illustrating one embodiment of asystem for data management in a solid-state storage device in accordancewith the present invention;

FIG. 1B is a schematic block diagram illustrating one embodiment of asystem for object management in a storage device in accordance with thepresent invention;

FIG. 1C is a schematic block diagram illustrating one embodiment of asystem for an in-server storage area network in accordance with thepresent invention;

FIG. 2A is a schematic block diagram illustrating one embodiment of anapparatus for object management in a storage device in accordance withthe present invention;

FIG. 2B is a schematic block diagram illustrating one embodiment of asolid-state storage device controller in a solid-state storage device inaccordance with the present invention;

FIG. 3 is a schematic block diagram illustrating one embodiment of asolid-state storage controller with a write data pipeline and a readdata pipeline in a solid-state storage device in accordance with thepresent invention;

FIG. 4A is a schematic block diagram illustrating one embodiment of abank interleave controller in the solid-state storage controller inaccordance with the present invention;

FIG. 4B is a schematic block diagram illustrating an alternateembodiment of a bank interleave controller in the solid-state storagecontroller in accordance with the present invention;

FIG. 5A is a schematic flow chart diagram illustrating one embodiment ofa method for managing data in a solid-state storage device using a datapipeline in accordance with the present invention;

FIG. 5B is a schematic flow chart diagram illustrating one embodiment ofa method for in-Server SAN in accordance with the present invention;

FIG. 6 is a schematic flow chart diagram illustrating another embodimentof a method for managing data in a solid-state storage device using adata pipeline in accordance with the present invention;

FIG. 7 is a schematic flow chart diagram illustrating an embodiment of amethod for managing data in a solid-state storage device using a bankinterleave in accordance with the present invention;

FIG. 8 is a schematic block diagram illustrating one embodiment of anapparatus for garbage collection in a solid-state storage device inaccordance with the present invention;

FIG. 9 is a schematic flow chart diagram illustrating one embodiment ofa method for garbage collection in a solid state storage device inaccordance with the present invention;

FIG. 10 is a schematic block diagram illustrating one embodiment of asystem for progressive RAID and front-end distributed RAID in accordancewith the present inventions; and

FIG. 11 is a schematic block diagram illustrating one embodiment of anapparatus for progressive RAID in accordance with the present invention.

FIG. 12 is a schematic block diagram illustrating one embodiment of anapparatus for updating a data segment using progressive RAID inaccordance with the present invention;

FIG. 13 is a schematic flow chart diagram illustrating an embodiment ofa method for managing data using progressive RAIDing in accordance withthe present invention;

FIG. 14 is a schematic flow chart diagram illustrating an embodiment ofa method for updating a data segment using progressive RAIDing inaccordance with the present invention;

FIG. 15 is a schematic block diagram illustrating one embodiment of anapparatus for front-end distributed RAID in accordance with the presentinvention;

FIG. 16 is a schematic flow chart diagram illustrating an embodiment ofa method for front-end distributed RAID in accordance with the presentinvention;

DETAILED DESCRIPTION OF THE INVENTION

Many of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more physical or logical blocks of computerinstructions which may, for instance, be organized as an object,procedure, or function. Nevertheless, the executables of an identifiedmodule need not be physically located together, but may comprisedisparate instructions stored in different locations which, when joinedlogically together, comprise the module and achieve the stated purposefor the module.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different storage devices, and may exist, atleast partially, merely as electronic signals on a system or network.Where a module or portions of a module are implemented in software, thesoftware portions are stored on one or more computer readable media.

Reference throughout this specification to “one embodiment,” “anembodiment,” or similar language means that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least one embodiment of the present invention. Thus,appearances of the phrases “in one embodiment,” “in an embodiment,” andsimilar language throughout this specification may, but do notnecessarily, all refer to the same embodiment.

Reference to a signal bearing medium may take any form capable ofgenerating a signal, causing a signal to be generated, or causingexecution of a program of machine-readable instructions on a digitalprocessing apparatus. A signal bearing medium may be embodied by atransmission line, a compact disk, digital-video disk, a magnetic tape,a Bernoulli drive, a magnetic disk, a punch card, flash memory,integrated circuits, or other digital processing apparatus memorydevice.

Furthermore, the described features, structures, or characteristics ofthe invention may be combined in any suitable manner in one or moreembodiments. In the following description, numerous specific details areprovided, such as examples of programming, software modules, userselections, network transactions, database queries, database structures,hardware modules, hardware circuits, hardware chips, etc., to provide athorough understanding of embodiments of the invention. One skilled inthe relevant art will recognize, however, that the invention may bepracticed without one or more of the specific details, or with othermethods, components, materials, and so forth. In other instances,well-known structures, materials, or operations are not shown ordescribed in detail to avoid obscuring aspects of the invention.

The schematic flow chart diagrams included herein are generally setforth as logical flow chart diagrams. As such, the depicted order andlabeled steps are indicative of one embodiment of the presented method.Other steps and methods may be conceived that are equivalent infunction, logic, or effect to one or more steps, or portions thereof, ofthe illustrated method. Additionally, the format and symbols employedare provided to explain the logical steps of the method and areunderstood not to limit the scope of the method. Although various arrowtypes and line types may be employed in the flow chart diagrams, theyare understood not to limit the scope of the corresponding method.Indeed, some arrows or other connectors may be used to indicate only thelogical flow of the method. For instance, an arrow may indicate awaiting or monitoring period of unspecified duration between enumeratedsteps of the depicted method. Additionally, the order in which aparticular method occurs may or may not strictly adhere to the order ofthe corresponding steps shown.

Solid-State Storage System

FIG. 1A is a schematic block diagram illustrating one embodiment of asystem 100 for data management in a solid-state storage device inaccordance with the present invention. The system 100 includes asolid-state storage device 102, a solid-state storage controller 104, awrite data pipeline 106, a read data pipeline 108, a solid-state storage110, a computer 112, a client 114, and a computer network 116, which aredescribed below.

The system 100 includes at least one solid-state storage device 102. Inanother embodiment, the system 100 includes two or more solid-statestorage devices 102. Each solid-state storage device 102 may includenon-volatile, solid-state storage 110, such as flash memory, nano randomaccess memory (“nano RAM or NRAM”), magneto-resistive RAM (“MRAM”),dynamic RAM (“DRAM”), phase change RAM (“PRAM”), etc. The solid-statestorage device 102 is described in more detail with respect to FIGS. 2and 3. The solid-state storage device 102 is depicted in a computer 112connected to a client 114 through a computer network 116. In oneembodiment, the solid-state storage device 102 is internal to thecomputer 112 and is connected using a system bus, such as a peripheralcomponent interconnect express (“PCI-e”) bus, a Serial AdvancedTechnology Attachment (“serial ATA”) bus, or the like. In anotherembodiment, the solid-state storage device 102 is external to thecomputer 112 and is connected, a universal serial bus (“USB”)connection, an Institute of Electrical and Electronics Engineers(“IEEE”) 1394 bus (“FireWire”), or the like. In other embodiments, thesolid-state storage device 102 is connected to the computer 112 using aperipheral component interconnect (“PCI”) express bus using externalelectrical or optical bus extension or bus networking solution such asInfiniband or PCI Express Advanced Switching (“PCIe-AS”), or the like.

In various embodiments, the solid-state storage device 102 may be in theform of a dual-inline memory module (“DIMM”), a daughter card, or amicro-module. In another embodiment, the solid-state storage device 102is an element within a rack-mounted blade. In another embodiment, thesolid state storage device 102 is contained within a package that isintegrated directly onto a higher level assembly (e.g. mother board, laptop, graphics processor). In another embodiment, individual componentscomprising the solid-state storage device 102 are integrated directlyonto a higher level assembly without intermediate packaging.

The solid-state storage device 102 includes one or more solid-statestorage controllers 104, each may include a write data pipeline 106 anda read data pipeline 108 and each includes a solid-state storage 110,which are described in more detail below with respect to FIGS. 2 and 3.

The system 100 includes one or more computers 112 connected to thesolid-state storage device 102. A computer 112 may be a host, a server,a storage controller of a storage area network (“SAN”), a workstation, apersonal computer, a laptop computer, a handheld computer, asupercomputer, a computer cluster, a network switch, router, orappliance, a database or storage appliance, a data acquisition or datacapture system, a diagnostic system, a test system, a robot, a portableelectronic device, a wireless device, or the like. In anotherembodiment, a computer 112 may be a client and the solid-state storagedevice 102 operates autonomously to service data requests sent from thecomputer 112. In this embodiment, the computer 112 and solid-statestorage device 102 may be connected using a computer network, systembus, or other communication means suitable for connection between acomputer 112 and an autonomous solid-state storage device 102.

In one embodiment, the system 100 includes one or more clients 114connected to one or more computer 112 through one or more computernetworks 116. A client 114 may be a host, a server, a storage controllerof a SAN, a workstation, a personal computer, a laptop computer, ahandheld computer, a supercomputer, a computer cluster, a networkswitch, router, or appliance, a database or storage appliance, a dataacquisition or data capture system, a diagnostic system, a test system,a robot, a portable electronic device, a wireless device, or the like.The computer network 116 may include the Internet, a wide area network(“WAN”), a metropolitan area network (“MAN”), a local area network(“LAN”), a token ring, a wireless network, a fiber channel network, aSAN, network attached storage (“NAS”), ESCON, or the like, or anycombination of networks. The computer network 116 may also include anetwork from the IEEE 802 family of network technologies, such Ethernet,token ring, WiFi, WiMax, and the like.

The computer network 116 may include servers, switches, routers,cabling, radios, and other equipment used to facilitate networkingcomputers 112 and clients 114. In one embodiment, the system 100includes multiple computers 112 that communicate as peers over acomputer network 116. In another embodiment, the system 100 includesmultiple solid-state storage devices 102 that communicate as peers overa computer network 116. One of skill in the art will recognize othercomputer networks 116 comprising one or more computer networks 116 andrelated equipment with single or redundant connection between one ormore clients 114 or other computer with one or more solid-state storagedevices 102 or one or more solid-state storage devices 102 connected toone or more computers 112. In one embodiment, the system 100 includestwo or more solid-state storage devices 102 connected through thecomputer network 116 to a client 114 without a computer 112.

Storage Controller-Managed Objects

FIG. 1B is a schematic block diagram illustrating one embodiment of asystem 101 for object management in a storage device in accordance withthe present invention. The system 101 includes one or more storagedevices 150, each with a storage controller 152 and one or more datastorage devices 154, and one or more requesting devices 155. The storagedevices 150 are networked together and coupled to one or more requestingdevices 155. The requesting device 155 sends object requests to astorage device 150 a. An object request may be a request to create anobject, a request to write data to an object, a request to read datafrom an object, a request to delete an object, a request to checkpointan object, a request to copy an object, and the like. One of skill inthe art will recognize other object requests.

In one embodiment, the storage controller 152 and data storage device154 are separate devices. In another embodiment, the storage controller152 and data storage device 154 are integrated into one storage device150. In another embodiment, a data storage device 154 is a solid-statestorage 110 and the storage controller 152 is a solid-state storagedevice controller 202. In other embodiments, a data storage device 154may be a hard disk drive, an optical drive, tape storage, or the like.In another embodiment, a storage device 150 may include two or more datastorage devices 154 of different types.

In one embodiment, the data storage device 154 is a solid-state storage110 and is arranged as an array of solid-state storage elements 216,218, 220. In another embodiment, the solid-state storage 110 is arrangedin two or more banks 214 a-n. Solid-state storage 110 is described inmore detail below with respect to FIG. 2B.

The storage devices 150 a-n may be networked together and act as adistributed storage device. The storage device 150 a coupled to therequesting device 155 controls object requests to the distributedstorage device. In one embodiment, the storage devices 150 andassociated storage controllers 152 manage objects and appear to therequesting device(s) 155 as a distributed object file system. In thiscontext, a parallel object file system is an example of a type ofdistributed object file system. In another embodiment, the storagedevices 150 and associated storage controllers 152 manage objects andappear to the requesting device(s) 155 as distributed object fileservers. In this context, a parallel object file server is an example ofa type of distributed object file server. In these and other embodimentsthe requesting device 155 may exclusively manage objects or participatein managing objects in conjunction with storage devices 150; thistypically does not limit the ability of storage devices 150 to fullymanage objects for other clients 114. In the degenerate case, eachdistributed storage device, distributed object file system anddistributed object file server can operate independently as a singledevice. The networked storage devices 150 a-n may operate as distributedstorage devices, distributed object file systems, distributed objectfile servers, and any combination thereof having images of one or moreof these capabilities configured for one or more requesting devices 155.For example, the storage devices 150 may be configured to operate asdistributed storage devices for a first requesting device 155 a, whileoperating as distributed storage devices and distributed object filesystems for requesting devices 155 b. Where the system 101 includes onestorage device 150 a, the storage controller 152 a of the storage device150 a manages objects may appear to the requesting device(s) 155 as anobject file system or an object file server.

In one embodiment where the storage devices 150 are networked togetheras a distributed storage device, the storage devices 150 serve as aredundant array of independent drives (“RAID”) managed by one or moredistributed storage controllers 152. For example, a request to write adata segment of an object results in the data segment being strippedacross the data storage devices 154 a-n with a parity stripe, dependingupon the RAID level. One benefit of such an arrangement is that such anobject management system may continue to be available when a singlestorage device 150 has a failure, whether of the storage controller 152,the data storage device 154, or other components of storage device 150.

When redundant networks are used to interconnect the storage devices 150and requesting devices 155, the object management system may continue tobe available in the presence of network failures as long as one of thenetworks remains operational. A system 101 with a single storage device150 a may also include multiple data storage devices 154 a and thestorage controller 152 a of the storage device 150 a may act as a RAIDcontroller and stripe the data segment across the data storage devices154 a of the storage device 150 a and may include a parity stripe,depending upon the RAID level.

In one embodiment, where the one or more storage devices 150 a-n aresolid-state storage devices 102 with a solid-state storage devicecontroller 202 and solid-state storage 110, the solid-state storagedevice(s) 102 may be configured in a DIMM configuration, daughter card,micro-module, etc. and reside in a computer 112. The computer 112 may bea server or similar device with the solid-state storage devices 102networked together and acting as distributed RAID controllers.Beneficially, the storage devices 102 may be connected using PCI-e,PCIe-AS, Infiniband or other high-performance bus, switched bus,networked bus, or network and may provide a very compact, highperformance RAID storage system with single or distributed solid-statestorage controllers 202 autonomously striping a data segment acrosssolid-state storage 110 a-n.

In one embodiment, the same network used by the requesting device 155 tocommunicate with storage devices 150 may be used by the peer storagedevice 150 a to communicate with peer storage devices 150 b-n toaccomplish RAID functionality. In another embodiment, a separate networkmay be used between the storage devices 150 for the purpose of RAIDing.In another embodiment, the requesting devices 155 may participate in theRAIDing process by sending redundant requests to the storage devices150. For example, requesting device 155 may send a first object writerequest to a first storage device 150 a and a second object writerequest with the same data segment to a second storage device 150 b toachieve simple mirroring.

With the ability for object handling within the storage device(s) 102,the storage controller(s) 152 uniquely have the ability to store onedata segment or object using one RAID level while another data segmentor object is stored using a different RAID level or without RAIDstriping. These multiple RAID groupings may be associated with multiplepartitions within the storage devices 150. RAID 0, RAID 1, RAID5, RAID6and composite RAID types 10, 50, 60, can be supported simultaneouslyacross a variety of RAID groups comprising data storage devices 154 a-n.One skilled in the art will recognize other RAID types andconfigurations that may also be simultaneously supported.

Also, because the storage controller(s) 152 operate autonomously as RAIDcontrollers, the RAID controllers can perform progressive RAIDing andcan transform objects or portions of objects striped across data storagedevices 154 with one RAID level to another RAID level without therequesting device 155 being affected, participating or even detectingthe change in RAID levels. In the preferred embodiment, progressing theRAID configuration from one level to another level may be accomplishedautonomously on an object or even a packet bases and is initiated by adistributed RAID control module operating in one of the storage devices150 or the storage controllers 152. Typically, RAID progression will befrom a higher performance and lower efficiency storage configurationsuch as RAID1 to a lower performance and higher storage efficiencyconfiguration such as RAID5 where the transformation is dynamicallyinitiated based on the frequency of access. But, one can see thatprogressing the configuration from RAID5 to RAID1 is also possible.Other processes for initiating RAID progression may be configured orrequested from clients or external agents such a storage systemmanagement server request. One of skill in the art will recognize otherfeatures and benefits of a storage device 102 with a storage controller152 that autonomously manages objects.

Solid-State Storage Device with In-Server San

FIG. 1C is a schematic block diagram illustrating one embodiment of asystem 103 for an in-server storage area network (“SAN”) in accordancewith the present invention. The system 103 includes a computer 112typically configured as a server (“server 112”). Each server 112includes one or more storage devices 150 where the server 112 andstorage devices 150 are each connected to a shared network interface156. Each storage device 150 includes a storage controller 152 andcorresponding data storage device 154. The system 103 includes clients114, 114 a, 114 b that are either internal or external to the servers112. The clients 114, 114 a, 114 b may communicate with each server 112and each storage device 150 through over one or more computer networks116, which are substantially similar to those described above.

The storage device 150 includes a DAS module 158, a NAS module 160, astorage communication module 162, an in-server SAN module 164, a commoninterface module 166, a proxy module 170, a virtual bus module 172, afront-end RAID module 174, and back-end RAID module 176, which aredescribed below. While the modules 158-176 are shown in a storage device150, all or a portion of each module 158-176 may be in the storagedevice 150, server 112, storage controller 152, or other location.

A server 112, as used in conjunction with in-server SAN, is a computerfunctioning as a server. The server 112 includes at least one serverfunction, such as a file server function, but may also include otherserver functions as well. The servers 112 may be part of a server farmand may service other clients 114. In other embodiments, the server 112may also be a personal computer, a workstation, or other computer thathouses storage devices 150. A server 112 may access one or more storagedevices 150 in the server 112 as direct attached storage (“DAS”), SANattached storage or network attached storage (“NAS”). Storagecontrollers 150 participating in an in-server SAN or NAS may be internalor external to the server 112.

In one embodiment, the in-server SAN apparatus includes a DAS module 158that configures at least a portion of the at least one data storagedevice 154 controlled by a storage controller 152 in a server 112 as aDAS device attached to the server 112 for servicing storage requestsfrom at least one client 114 to the server 112. In one embodiment, afirst data storage device 154 a is configured as a DAS to the firstserver 112 a while also being configured as an in-server SAN storagedevice to the first server 112 a. In another embodiment, the first datastorage device 154 a is partitioned so one partition is a DAS and theother is an in-server SAN. In another embodiment, at least a portion ofstorage space within the first data storage device 154 a is configuredas a DAS to the first server 112 a and the same portion of storage spaceon the first data storage device 154 a is configured as an in-server SANto the first server 112 a.

In another embodiment, the in-server SAN apparatus includes a NAS module160 that configures a storage controller 152 as a NAS device for atleast one client 114 and services file requests from the client 114. Thestorage controller 152 may be also configured as an in-server SAN devicefor the first server 112 a. The storage devices 150 may directly connectto the computer network 116 through the shared network interface 156independent from the server 112 in which the storage device 150 resides.

In one elemental form, an apparatus for in-server SAN includes a firststorage controller 152 a within a first server 112 a where the firststorage controller 152 a controls at least one storage device 154 a. Thefirst server 112 a includes a network interface 156 shared by the firstserver 112 a and the first storage controller 152 a. The in-server SANapparatus includes a storage communication module 162 that facilitatescommunication between the first storage controller 152 a and at leastone device external to the first server 112 a such that thecommunication between the first storage controller 152 a and theexternal device is independent from the first server 112 a. The storagecommunication module 162 may allow the first storage controller 152 a toindependently access the network interface 156 a for externalcommunication. In one embodiment, the storage communication module 162accesses a switch in the network interface 156 a to direct networktraffic between the first storage controller 152 a and external devices.

The in-server SAN apparatus also includes an in-server SAN module 164that services a storage request using one or both of a network protocoland a bus protocol. The in-server SAN module 164 services the storagerequest independent from the first server 112 a and the service requestis received from an internal or external client 114, 114 a.

In one embodiment, the device external to the first server 112 a is asecond storage controller 152 b. The second storage controller 152 bcontrols at least one data storage device 154 b. The in-server SANmodule 164 services the storage request using communication through thenetwork interface 156 a and between the first and second storagecontrollers 152 a, 152 b independent of the first server 112 a. Thesecond storage controller 152 b may be within a second server 112 b orwithin some other device.

In another embodiment, the device external to the first server 112 a isa client 114 and the storage request originates with the external client114 where the first storage controller is configured as at least part ofa SAN and the in-server SAN module 164 services the storage requestthrough the network interface 156 a independent of the first server 112a. The external client 114 may be in the second server 112 b or may beexternal to the second server 112 b. In one embodiment, the in-serverSAN module 164 can service storage requests from the external client 114even when the first server 112 a is unavailable.

In another embodiment, the client 114 a originating the storage requestis internal to the first server 112 a where the first storage controller152 a is configured as at least part of a SAN and the in-server SANmodule 164 services the storage request through one or more of thenetwork interface 156 a and system bus.

Traditional SAN configurations allow a storage device remote from aserver 112 to be accessed as if the storage device resides within theserver 112 as direct attached storage (“DAS”) so that the storage deviceappears as a block storage device. Typically, a storage device connectedas a SAN requires a SAN protocol, such as fiber channel, Internet smallcomputer system interface (“iSCSI”), HyperSCSI, Fiber Connectivity(“FICON”), Advanced Technology Attachment (“ATA”) over Ethernet, etc.In-server SAN includes a storage controller 152 inside a server 112while still allowing network connection between the storage controller152 a and a remote storage controller 152 b or an external client 114using a network protocol and/or a bus protocol.

Typically, SAN protocols are a form of network protocol and more networkprotocols are emerging, such as Infiniband that would allow a storagecontroller 150 a, and associated data storage devices 154 a, to beconfigured as a SAN and communicate with an external client 114 orsecond storage controller 152 b. In another example, a first storagecontroller 152 a may communicate with an external client 114 or secondstorage controller 152 b using Ethernet.

A storage controller 152 may communicate over a bus with internalstorage controllers 152 or clients 114 a. For example, a storagecontroller 152 may communicate over a bus using PCI-e that may supportPCI Express Input/Output Virtualization (“PCIe-IOV”). Other emerging busprotocols allow a system bus to extend outside a computer or server 112and would allow a storage controller 152 a to be configured as a SAN.One such bus protocol is PCIe-AS. The present invention is not limitedto simply SAN protocols, but may also take advantage of the emergingnetwork and bus protocols to service storage requests. An externaldevice, either in the form of a client 114 or external storagecontroller 152 b, may communicate over an extended system bus or acomputer network 116. A storage request, as used herein, includesrequests to write data, read data, erase data, query data, etc. and mayinclude object data, metadata, and management requests as well as blockdata requests.

A traditional server 112 typically has a root complex that controlsaccess to devices within the server 112. Typically, this root complex ofthe server 112 owns the network interface 156 such so any communicationthrough the network interface 156 is controlled by the server 112.However, in the preferred embodiment of the in-server SAN apparatus, thestorage controller 152 is able to access the network interface 156independently so that clients 114 may communicate directly with one ormore of the storage controllers 152 a in the first server 112 a forminga SAN or so that one or more first storage controllers 152 a may benetworked together with a second storage controller 152 b or otherremote storage controllers 152 to form a SAN. In the preferredembodiment, devices remote from the first server 112 a may access thefirst server 112 a or the first storage controller 152 a through asingle, shared network address. In one embodiment, the in-server SANapparatus includes a common interface module 166 that configures thenetwork interface 156, the storage controller 152, and the server 112such that the server 112 and the storage controller 152 are accessibleusing a shared network address.

In other embodiments, the server 112 includes two or more networkinterfaces 156. For example, the server 112 may communicate over onenetwork interface 156 while the storage device 150 may communicate overanother interface. In another example, the server 112 includes multiplestorage devices 150, each with a network interface 156. One of skill inthe art will recognize other configurations of a server 112 with one ormore storage devices 150 and one or more network interfaces 156 whereone or more of the storage devices 150 access a network interface 156independent of the server 112. One of skill in the art will alsorecognize how these various configurations may be extended to supportnetwork redundancy and improve availability.

Advantageously, the in-server SAN apparatus eliminates much of thecomplexity and expense of a traditional SAN. For example, a typical SANrequires servers 112 with external storage controllers 152 andassociated data storage devices 154. This takes up additional space in arack and requires cabling, switches, etc. The cabling, switching,another other overhead required to configure a traditional SAN takespace, degrade bandwidth, and are expensive. The in-server SAN apparatusallows the storage controllers 152 and associated storage 154 to fit ina server 112 form factor, thus reducing required space and costing less.In-server SAN also allows connection using relatively fast communicationover internal and external high-speed data buses.

In one embodiment, the storage device 150 is a solid-state storagedevice 102, the storage controller 152 is a solid-state storagecontroller 104, and the data storage device 154 is a solid-state storage110. This embodiment is advantageous because of the speed of solid-statestorage device 102 as described herein. In addition, the solid-statestorage device 102 may be configured in a DIMM which may convenientlyfit in a server 112 and require a small amount of space.

The one or more internal clients 114 a in the server 112 may alsoconnect to the computer network 116 through the server's networkinterface 156 and the client's connection is typically controlled by theserver 112. This has several advantages. Clients 114 a may locally andremotely access the storage devices 150 directly and may initiate alocal or remote direct memory access (“DMA,” “RDMA”) data transferbetween the memory of a client 114 a and a storage device 150.

In another embodiment, clients 114, 114 a within or external to a server112 may act as file servers to clients 114 through one or more networks116 while utilizing locally attached storage devices 150 as DAS devices,network attached storage devices 150, network attached solid-statestorages 102 devices participating as part of in-server SANs, externalSANs, and hybrid SANs. A storage device 150 may participate in a DAS,in-server-SAN, SAN, NAS, etc, simultaneously and in any combination.Additionally, each storage device 150 may be partitioned in such a waythat a first partition makes the storage device 150 available as a DAS,a second partition makes the storage device 150 available as an elementin an in-server-SAN, a third partition makes the storage device 150available as a NAS, a fourth partition makes the storage device 150available as an element in a SAN, etc. Similarly, the storage device 150may be partitioned consistent with security and access controlrequirements. One of skill in the art will recognize that any number ofcombinations and permutations of storage devices, virtual storagedevices, storage networks, virtual storage networks, private storage,shared storage, parallel file systems, parallel object file systems,block storage devices, object storage devices, storage appliances,network appliances, and the like may be constructed and supported.

In addition, by directly connecting to the computer network 116, thestorage devices 150 can communicate with each other and can act as anin-server SAN. Clients 114 a in the servers 112 and clients 114connected through the computer network 116 may access the storagedevices 150 as a SAN. By moving the storage devices 150 into the servers112 and having the ability to configure the storage devices 150 as aSAN, the server 112/storage device 150 combination eliminates the needin conventional SANs for dedicated storage controllers, fiber channelnetworks, and other equipment. The in-server SAN system 103 has theadvantage of enabling the storage device 150 to share common resourcessuch as power, cooling, management, and physical space with the client114 and computer 112. For example, storage devices 150 may fill emptyslots of servers 112 and provide all the performance capabilities,reliability and availability of a SAN or NAS. One of skill in the artwill recognize other features and benefits of an in-server SAN system103.

In another configuration, multiple in-server-SAN storage devices 150 aare collocated within a single server 112 a infrastructure. In oneembodiment, the server 112 a is comprised of one or more internal bladedserver clients 114 a interconnected using PCI-express IOV without anexternal network interface 156, external client 114, 114 b or externalstorages device 150 b.

In addition, in-server SAN storage device 150 may communicate throughone or more computer networks 116 with peer storage devices 150 that arelocated in a computer 112 (per FIG. 1A), or are connected directly tothe computer network 116 without a computer 112 to form a hybrid SANwhich has all the capabilities of both SAN and in-server SAN. Thisflexibility has the benefit of simplifying extensibility and migrationbetween a variety of possible solid-state storage networkimplementations. One skilled in the art will recognize othercombinations, configurations, implementations, and architectures forlocating and interconnecting solid-state controllers 104.

Where the network interface 156 a can be controlled by only one agentoperating within the server 112 a, a link setup module 168 operatingwithin that agent can set up communication paths between internalclients 114 a and storage devices 150 a/first storage controllers 152 athrough network interface 156 a to external storage devices 150 b andclients 114, 114 b. In a preferred embodiment, once the communicationpath is established, the individual internal storage devices 150 a andinternal clients 114 a are able to establish and manage their owncommand queues and transfer both commands and data through networkinterface 156 a to external storage devices 150 b and clients 114, 114 bin either direction, directly and through RDMA independent of the proxyor agent controlling the network interface 156 a. In one embodiment, thelink setup module 168 establishes the communication links during aninitialization process, such as a startup or initialization of hardware.

In another embodiment, a proxy module 170 directs at least a portion ofcommands used in servicing a storage request through the first server112 a while at least data, and possibly other commands, associated withthe storage request are communicated between the first storagecontroller and the external storage device independent of the firstserver. In another embodiment, the proxy module 170 forwards commands ordata in behalf of the internal storage devices 150 a and clients 114 a.

In one embodiment, the first server 112 a includes one or more serverswithin the first server 112 a and includes a virtual bus module 172 thatallows the one or more servers in the first server 112 a toindependently access one or more storage controllers 152 a throughseparate virtual buses. The virtual buses may be established using anadvanced bus protocol such as PCIe-IOV. Network interfaces 156 asupporting IOV may allow the one or more servers and the one or morestorage controllers to independently control the one or more networkinterfaces 156 a.

In various embodiments, the in-server SAN apparatus allows two or morestorage devices 150 to be configured in a RAID. In one embodiment, thein-server SAN apparatus includes a front-end RAID module 174 thatconfigures two or more storage controllers 152 as a RAID. Where astorage request from a client 114, 114 a includes a request to storedata, the front-end RAID module 174 services the storage request bywriting the data to the RAID consistent with the particular implementedRAID level. A second storage controller 152 may be located either in thefirst server 112 a or external to the first server 112 a. The front-endRAID module 174 allows RAIDing of storage controllers 152 such that thestorage controllers 152 are visible to the client 114, 114 a sending thestorage request. This allows striping and parity information to bemanaged by a storage controller 152 designated as master or by theclient 114, 114 a.

In another embodiment, the in-server SAN apparatus includes a back-endRAID module 176 that configures two or more data storage devices 154controlled by a storage controller as a RAID. Where the storage requestfrom the client comprises a request to store data, the back-end RAIDmodule 176 services the storage request by writing the data to the RAIDconsistent with an implemented RAID level such that the storage devices154 configured as a RAID are accessed by the client 114, 114 a as asingle data storage device 154 controlled by the first storagecontroller 152. This RAID implementation allows RAIDing of the datastorage devices 154 controlled by a storage controller 152 in a way thatthe RAIDing is transparent to any client 114, 114 a accessing the datastorage devices 154. In another embodiment, both front-end RAID andback-end RAID are implemented to have multi-level RAID. One of skill inthe art will recognize other ways to RAID the storage devices 152consistent with the solid-state storage controller 104 and associatedsolid-state storage 110 described herein.

Apparatus for Storage Controller-Managed Objects

FIG. 2A is a schematic block diagram illustrating one embodiment of anapparatus 200 for object management in a storage device in accordancewith the present invention. The apparatus 200 includes a storagecontroller 152 with an object request receiver module 260, a parsingmodule 262, a command execution module 264, an object index module 266,an object request queuing module 268, a packetizer 302 with a messagesmodule 270, and an object index reconstruction module 272, which aredescribed below.

The storage controller 152 is substantially similar to the storagecontroller 152 described in relation to the system 101 of FIG. 1B andmay be a solid-state storage device controller 202 described in relationto FIG. 2. The apparatus 200 includes an object request receiver module260 that receives an object request from one or more requesting devices155. For example, for a store object data request, the storagecontroller 152 stores the data segment as a data packet in a datastorage device 154 coupled to the storage controller 152. The objectrequest is typically directed at a data segment stored or to be storedin one or more object data packets for an object managed by the storagecontroller 152. The object request may request that the storagecontroller 152 create an object to be later filled with data throughlater object request which may utilize a local or remote direct memoryaccess (“DMA,” “RDMA”) transfer.

In one embodiment, the object request is a write request to write all orpart of an object to a previously created object. In one example, thewrite request is for a data segment of an object. The other datasegments of the object may be written to the storage device 150 or toother storage devices. In another example, the write request is for anentire object. In another example, the object request is to read datafrom a data segment managed by the storage controller 152. In yetanother embodiment, the object request is a delete request to delete adata segment or object.

Advantageously, the storage controller 152 can accept write requeststhat do more than write a new object or append data to an existingobject. For example, a write request received by the object requestreceiver module 260 may include a request to add data ahead of datastored by the storage controller 152, to insert data into the storeddata, or to replace a segment of data. The object index maintained bythe storage controller 152 provides the flexibility required for thesecomplex write operations that is not available in other storagecontrollers, but is currently available only outside of storagecontrollers in file systems of servers and other computers.

The apparatus 200 includes a parsing module 262 that parses the objectrequest into one or more commands. Typically, the parsing module 262parses the object request into one or more buffers. For example, one ormore commands in the object request may be parsed into a command buffer.Typically the parsing module 262 prepares an object request so that theinformation in the object request can be understood and executed by thestorage controller 152. One of skill in the art will recognize otherfunctions of a parsing module 262 that parses an object request into oneor more commands.

The apparatus 200 includes a command execution module 264 that executesthe command(s) parsed from the object request. In one embodiment, thecommand execution module 264 executes one command. In anotherembodiment, the command execution module 264 executes multiple commands.Typically, the command execution module 264 interprets a command parsedfrom the object request, such as a write command, and then creates,queues, and executes subcommands. For example, a write command parsedfrom an object request may direct the storage controller 152 to storemultiple data segments. The object request may also include requiredattributes such as encryption, compression, etc. The command executionmodule 264 may direct the storage controller 152 to compress the datasegments, encrypt the data segments, create one or more data packets andassociated headers for each data packet, encrypt the data packets with amedia encryption key, add error correcting code, and store the datapackets a specific location. Storing the data packets at a specificlocation and other subcommands may also be broken down into other lowerlevel subcommands. One of skill in the art will recognize other waysthat the command execution module 264 can execute one or more commandsparsed from an object request.

The apparatus 200 includes an object index module 266 that creates anobject entry in an object index in response to the storage controller152 creating an object or storing the data segment of the object.Typically, the storage controller 152 creates a data packet from thedata segment and the location of where the data packet is stored isassigned at the time the data segment is stored. Object metadatareceived with a data segment or as part of an object request may bestored in a similar way.

The object index module 266 creates an object entry into an object indexat the time the data packet is stored and the physical address of thedata packet is assigned. The object entry includes a mapping between alogical identifier of the object and one or more physical addressescorresponding to where the storage controller 152 stored one or moredata packets and any object metadata packets. In another embodiment, theentry in the object index is created before the data packets of theobject are stored. For example, if the storage controller 152 determinesa physical address of where the data packets are to be stored earlier,the object index module 266 may create the entry in the object indexearlier.

Typically, when an object request or group of object requests results inan object or data segment being modified, possibly during aread-modify-write operation, the object index module 266 updates anentry in the object index corresponding to the modified object. In oneembodiment, the object index creates a new object and a new entry in theobject index for the modified object. Typically, where only a portion ofan object is modified, the object includes modified data packets andsome data packets that remain unchanged. In this case, the new entryincludes a mapping to the unchanged data packets as where they wereoriginally written and to the modified objects written to a newlocation.

In another embodiment, where the object request receiver module 260receives an object request that includes a command that erases a datablock or other object elements, the storage controller 152 may store atleast one packet such as an erase packet that includes informationincluding a reference to the object, relationship to the object, and thesize of the data block erased. Additionally, it may further indicatethat the erased object elements are filled with zeros. Thus, the eraseobject request can be used to emulate actual memory or storage that iserased and actually has a portion of the appropriate memory/storageactually stored with zeros in the cells of the memory/storage.

Beneficially, creating an object index with entries indicating mappingbetween data segments and metadata of an object allows the storagecontroller 152 to autonomously handle and manage objects. Thiscapability allows a great amount of flexibility for storing data in thestorage device 150. Once the index entry for the object is created,subsequent object requests regarding the object can be servicedefficiently by the storage controller 152.

In one embodiment, the storage controller 152 includes an object requestqueuing module 268 that queues one or more object requests received bythe object request receiver module 260 prior to parsing by the parsingmodule 262. The object request queuing module 268 allows flexibilitybetween when an object request is received and when it is queued.

In another embodiment, the storage controller 152 includes a packetizer302 that creates one or more data packets from the one or more datasegments where the data packets are sized for storage in the datastorage device 154. The packetizer 302 is described below in more detailwith respect to FIG. 3. The packetizer 302 includes, in one embodiment,a messages module 270 that creates a header for each packet. The headerincludes a packet identifier and a packet length. The packet identifierrelates the packet to the object for which the packet was formed.

In one embodiment, each packet includes a packet identifier that isself-contained in that the packet identifier contains adequateinformation to identify the object and relationship within the object ofthe object elements contained within the packet. However, a moreefficient preferred embodiment is to store packets in containers.

A container is a data construct that facilitates more efficient storageof packets and helps establish relationships between an object and datapackets, metadata packets, and other packets related to the object thatare stored within the container. Note that the storage controller 152typically treats object metadata received as part of an object and datasegments in a similar manner. Typically “packet” may refer to a datapacket comprising data, a metadata packet comprising metadata, oranother packet of another packet type. An object may be stored in one ormore containers and a container typically includes packets for no morethan one unique object. An object may be distributed between multiplecontainers. Typically a container is stored within a single logicalerase block (storage division) and is typically never split betweenlogical erase blocks.

A container, in one example, may be split between two or morelogical/virtual pages. A container is identified by a container labelthat associates that container with an object. A container may containzero to many packets and the packets within a container are typicallyfrom one object. A packet may be of many object element types, includingobject attribute elements, object data elements, object index elements,and the like. Hybrid packets may be created that include more than oneobject element type. Each packet may contain zero to many elements ofthe same element type. Each packet within a container typically containsa unique identifier that identifies the relationship to the object.

Each packet is associated with one container. In a preferred embodiment,containers are limited to an erase block so that at or near thebeginning of each erase block a container packet can be found. Thishelps limit data loss to an erase block with a corrupted packet header.In this embodiment, if the object index is unavailable and a packetheader within the erase block is corrupted, the contents from thecorrupted packet header to the end of the erase block may be lostbecause there is possibly no reliable mechanism to determine thelocation of subsequent packets. In another embodiment, a more reliableapproach is to have a container limited to a page boundary. Thisembodiment requires more header overhead. In another embodiment,containers can flow across page and erase block boundaries. Thisrequires less header overhead but a larger portion of data may be lostif a packet header is corrupted. For these several embodiments it isexpected that some type of RAID is used to further ensure dataintegrity.

In one embodiment, the apparatus 200 includes an object indexreconstruction module 272 that that reconstructs the entries in theobject index using information from packet headers stored in the datastorage device 154. In one embodiment, the object index reconstructionmodule 272 reconstructs the entries of the object index by readingheaders to determine the object to which each packet belongs andsequence information to determine where in the object the data ormetadata belongs. The object index reconstruction module 272 usesphysical address information for each packet and timestamp or sequenceinformation to create a mapping between the physical locations of thepackets and the object identifier and data segment sequence. Timestampor sequence information is used by the object index reconstructionmodule 272 to replay the sequence of changes made to the index andthereby typically reestablish the most recent state.

In another embodiment, the object index reconstruction module 272locates packets using packet header information along with containerpacket information to identify physical locations of the packets, objectidentifier, and sequence number of each packet to reconstruct entries inthe object index. In one embodiment, erase blocks are time stamped orgiven a sequence number as packets are written and the timestamp orsequence information of an erase block is used along with informationgathered from container headers and packet headers to reconstruct theobject index. In another embodiment, timestamp or sequence informationis written to an erase block when the erase block is recovered.

Where the object index is stored in volatile memory, an error, loss ofpower, or other problem causing the storage controller 152 to shut downwithout saving the object index could be a problem if the object indexcannot be reconstructed. The object index reconstruction module 272allows the object index to be stored in volatile memory allowing theadvantages of volatile memory, such as fast access. The object indexreconstruction module 272 allows quick reconstruction of the objectindex autonomously without dependence on a device external to thestorage device 150.

In one embodiment, the object index in volatile memory is storedperiodically in a data storage device 154. In a particular example, theobject index, or “index metadata,” is stored periodically in asolid-state storage 110. In another embodiment, the index metadata isstored in a solid-state storage 110 n separate from solid-state storage110 a-110 n−1 storing packets. The index metadata is managedindependently from data and object metadata transmitted from arequesting device 155 and managed by the storage controller152/solid-state storage device controller 202. Managing and storingindex metadata separate from other data and metadata from an objectallows efficient data flow without the storage controller152/solid-state storage device controller 202 unnecessarily processingobject metadata.

In one embodiment, where an object request received by the objectrequest receiver module 260 includes a write request, the storagecontroller 152 receives one or more data segments of an object frommemory of a requesting device 155 as a local or remote direct memoryaccess (“DMA,” “RDMA”) operation. In a preferred example, the storagecontroller 152 pulls data from the memory of the requesting device 155in one or more DMA or RDMA operations. In another example, therequesting device 155 pushes the data segment(s) to the storagecontroller 152 in one or more DMA or RDMA operations. In anotherembodiment, where the object request includes a read request, thestorage controller 152 transmits one or more data segments of an objectto the memory of the requesting device 155 in one or more DMA or RDMAoperations. In a preferred example, the storage controller 152 pushesdata to the memory of the requesting device 155 in one or more DMA orRDMA operations. In another example, the requesting device 155 pullsdata from the storage controller 152 in one or more DMA or RDMAoperations. In another example, the storage controller 152 pulls objectcommand request sets from the memory of the requesting device 155 in oneor more DMA or RDMA operations. In another example, the requestingdevice 155 pushes object command request sets to the storage controller152 in one or more DMA or RDMA operations.

In one embodiment, the storage controller 152 emulates block storage andan object communicated between the requesting device 155 and the storagecontroller 152 comprises one or more data blocks. In one embodiment, therequesting device 155 includes a driver so that the storage device 150appears as a block storage device. For example, the requesting device152 may send a block of data of a certain size along with a physicaladdress of where the requesting device 155 wants the data block stored.The storage controller 152 receives the data block and uses the physicalblock address transmitted with the data block or a transformation of thephysical block address as an object identifier. The storage controller152 then stores the data block as an object or data segment of an objectby packetizing the data block and storing the data block at will. Theobject index module 266 then creates an entry in the object index usingthe physical block-based object identifier and the actual physicallocation where the storage controller 152 stored the data packetscomprising the data from the data block.

In another embodiment, the storage controller 152 emulates block storageby accepting block objects. A block object may include one or more datablocks in a block structure. In one embodiment, the storage controller152 treats the block object as any other object. In another embodiment,an object may represent an entire block device, partition of a blockdevice, or some other logical or physical sub-element of a block deviceincluding a track, sector, channel, and the like. Of particular note isthe ability to remap a block device RAID group to an object supporting adifferent RAID construction such as progressive RAID. One skilled in theart will recognize other mappings of traditional or future block devicesto objects.

Solid-State Storage Device

FIG. 2B is a schematic block diagram illustrating one embodiment 201 ofa solid-state storage device controller 202 that includes a solid-statestorage device 102 in accordance with the present invention. Thesolid-state storage device controller 202 may include a number ofsolid-state storage controllers 0-N 104 a-n, each controllingsolid-state storage 110. In the depicted embodiment, two solid-statecontrollers are shown: solid-state controller 0 104 a and solid-statestorage controller N 104 n, and each controls solid-state storage 110a-n. In the depicted embodiment, solid-state storage controller 0 104 acontrols a data channel so that the attached solid-state storage 110 astores data. Solid-state storage controller N 104 n controls an indexmetadata channel associated with the stored data and the associatedsolid-state storage 110 n stores index metadata. In an alternateembodiment, the solid-state storage device controller 202 includes asingle solid-state controller 104 a with a single solid-state storage110 a. In another embodiment, there are a plurality of solid-statestorage controllers 104 a-n and associated solid-state storage 110 a-n.In one embodiment, one or more solid state controllers 104 a-104 n−1,coupled to their associated solid-state storage 110 a-110 n−1, controldata while at least one solid-state storage controller 104 n, coupled toits associated solid-state storage 110 n, controls index metadata.

In one embodiment, at least one solid-state controller 104 isfield-programmable gate array (“FPGA”) and controller functions areprogrammed into the FPGA. In a particular embodiment, the FPGA is aXilinx® FPGA. In another embodiment, the solid-state storage controller104 comprises components specifically designed as a solid-state storagecontroller 104, such as an application-specific integrated circuit(“ASIC”) or custom logic solution. Each solid-state storage controller104 typically includes a write data pipeline 106 and a read datapipeline 108, which are describe further in relation to FIG. 3. Inanother embodiment, at least one solid-state storage controller 104 ismade up of a combination FPGA, ASIC, and custom logic components.

Solid-State Storage

The solid state storage 110 is an array of non-volatile solid-statestorage elements 216, 218, 220, arranged in banks 214, and accessed inparallel through a bi-directional storage input/output (“I/O”) bus 210.The storage I/O bus 210, in one embodiment, is capable of unidirectionalcommunication at any one time. For example, when data is being writtento the solid-state storage 110, data cannot be read from the solid-statestorage 110. In another embodiment, data can flow both directionssimultaneously. However bi-directional, as used herein with respect to adata bus, refers to a data pathway that can have data flowing in onlyone direction at a time, but when data flowing one direction on thebi-directional data bus is stopped, data can flow in the oppositedirection on the bi-directional data bus.

A solid-state storage element (e.g. SSS 0.0 216 a) is typicallyconfigured as a chip (a package of one or more dies) or a die on acircuit board. As depicted, a solid-state storage element (e.g. 216 a)operates independently or semi-independently of other solid-statestorage elements (e.g. 218 a) even if these several elements arepackaged together in a chip package, a stack of chip packages, or someother package element. As depicted, a column of solid-state storageelements 216, 218, 220 is designated as a bank 214. As depicted, theremay be “n” banks 214 a-n and “m” solid-state storage elements 216 a-m,218 a-m, 220 a-m per bank in an array of n×m solid-state storageelements 216, 218, 220 in a solid-state storage 110. In one embodiment,a solid-state storage 110 a includes twenty solid-state storage elements216, 218, 220 per bank 214 with eight banks 214 and a solid-statestorage 110 n includes 2 solid-state storage elements 216, 218 per bank214 with one bank 214. In one embodiment, each solid-state storageelement 216, 218, 220 is comprised of a single-level cell (“SLC”)devices. In another embodiment, each solid-state storage element 216,218, 220 is comprised of multi-level cell (“MLC”) devices.

In one embodiment, solid-state storage elements for multiple banks thatshare a common storage I/O bus 210 a row (e.g. 216 b, 218 b, 220 b) arepackaged together. In one embodiment, a solid-state storage element 216,218, 220 may have one or more dies per chip with one or more chipsstacked vertically and each die may be accessed independently. Inanother embodiment, a solid-state storage element (e.g. SSS 0.0 216 a)may have one or more virtual dies per die and one or more dies per chipand one or more chips stacked vertically and each virtual die may beaccessed independently. In another embodiment, a solid-state storageelement SSS 0.0 216 a may have one or more virtual dies per die and oneor more dies per chip with some or all of the one or more dies stackedvertically and each virtual die may be accessed independently.

In one embodiment, two dies are stacked vertically with four stacks pergroup to form eight storage elements (e.g. SSS 0.0-SSS 0.8) 216 a-220 a,each in a separate bank 214 a-n. In another embodiment, 20 storageelements (e.g. SSS 0.0-SSS 20.0) 216 form a virtual bank 214 a so thateach of the eight virtual banks has 20 storage elements (e.g. SSS0.0-SSS20.8) 216, 218, 220. Data is sent to the solid-state storage 110 overthe storage I/O bus 210 to all storage elements of a particular group ofstorage elements (SSS 0.0-SSS 0.8) 216 a, 218 a, 220 a. The storagecontrol bus 212 a is used to select a particular bank (e.g. Bank-0 214a) so that the data received over the storage I/O bus 210 connected toall banks 214 is written just to the selected bank 214 a.

In a preferred embodiment, the storage I/O bus 210 is comprised of oneor more independent I/O buses (“IIOBa-m” comprising 210 a.a-m, 210n.a-m) wherein the solid-state storage elements within each row shareone of the independent I/O buses accesses each solid-state storageelement 216, 218, 220 in parallel so that all banks 214 are accessedsimultaneously. For example, one channel of the storage I/O bus 210 mayaccess a first solid-state storage element 216 a, 218 a, 220 a of eachbank 214 a-n simultaneously. A second channel of the storage I/O bus 210may access a second solid-state storage element 216 b, 218 b, 220 b ofeach bank 214 a-n simultaneously. Each row of solid-state storageelement 216, 218, 220 is accessed simultaneously. In one embodiment,where solid-state storage elements 216, 218, 220 are multi-level(physically stacked), all physical levels of the solid-state storageelements 216, 218, 220 are accessed simultaneously. As used herein,“simultaneously” also includes near simultaneous access where devicesare accessed at slightly different intervals to avoid switching noise.Simultaneously is used in this context to be distinguished from asequential or serial access wherein commands and/or data are sentindividually one after the other.

Typically, banks 214 a-n are independently selected using the storagecontrol bus 212. In one embodiment, a bank 214 is selected using a chipenable or chip select. Where both chip select and chip enable areavailable, the storage control bus 212 may select one level of amulti-level solid-state storage element 216, 218, 220. In otherembodiments, other commands are used by the storage control bus 212 toindividually select one level of a multi-level solid-state storageelement 216, 218, 220. Solid-state storage elements 216, 218, 220 mayalso be selected through a combination of control and of addressinformation transmitted on storage I/O bus 210 and the storage controlbus 212.

In one embodiment, each solid-state storage element 216, 218, 220 ispartitioned into erase blocks and each erase block is partitioned intopages. A typical page is 2000 bytes (“2 kB”). In one example, asolid-state storage element (e.g. SSS0.0) includes two registers and canprogram two pages so that a two-register solid-state storage element216, 218, 220 has a capacity of 4 kB. A bank 214 of 20 solid-statestorage elements 216, 218, 220 would then have an 80 kB capacity ofpages accessed with the same address going out the channels of thestorage I/O bus 210.

This group of pages in a bank 214 of solid-state storage elements 216,218, 220 of 80 kB may be called a virtual page. Similarly, an eraseblock of each storage element 216 a-m of a bank 214 a may be grouped toform a virtual erase block. In a preferred embodiment, an erase block ofpages within a solid-state storage element 216, 218, 220 is erased whenan erase command is received within a solid-state storage element 216,218, 220. Whereas the size and number of erase blocks, pages, planes, orother logical and physical divisions within a solid-state storageelement 216, 218, 220 are expected to change over time with advancementsin technology, it is to be expected that many embodiments consistentwith new configurations are possible and are consistent with the generaldescription herein.

Typically, when a packet is written to a particular location within asolid-state storage element 216, 218, 220, wherein the packet isintended to be written to a location within a particular page which isspecific to a of a particular erase block of a particular element of aparticular bank, a physical address is sent on the storage I/O bus 210and followed by the packet. The physical address contains enoughinformation for the solid-state storage element 216, 218, 220 to directthe packet to the designated location within the page. Since all storageelements in a row of storage elements (e.g. SSS 0.0-SSS 0.N 216 a, 218a, 220 a) are accessed simultaneously by the appropriate bus within thestorage I/O bus 210 a.a, to reach the proper page and to avoid writingthe data packet to similarly addressed pages in the row of storageelements (SSS 0.0-SSS 0.N 216 a, 218 a, 220 a), the bank 214 a thatincludes the solid-state storage element SSS 0.0 216 a with the correctpage where the data packet is to be written is simultaneously selectedby the storage control bus 212.

Similarly, a read command traveling on the storage I/O bus 210 requiresa simultaneous command on the storage control bus 212 to select a singlebank 214 a and the appropriate page within that bank 214 a. In apreferred embodiment, a read command reads an entire page, and becausethere are multiple solid-state storage elements 216, 218, 220 inparallel in a bank 214, an entire virtual page is read with a readcommand. However, the read command may be broken into subcommands, aswill be explained below with respect to bank interleave. A virtual pagemay also be accessed in a write operation.

An erase block erase command may be sent out to erase an erase blockover the storage I/O bus 210 with a particular erase block address toerase a particular erase block. Typically, an erase block erase commandmay be sent over the parallel paths of the storage I/O bus 210 to erasea virtual erase block, each with a particular erase block address toerase a particular erase block. Simultaneously a particular bank (e.g.bank-0 214 a) is selected over the storage control bus 212 to preventerasure of similarly addressed erase blocks in all of the banks (banks1-N 214 b-n). Other commands may also be sent to a particular locationusing a combination of the storage I/O bus 210 and the storage controlbus 212. One of skill in the art will recognize other ways to select aparticular storage location using the bi-directional storage I/O bus 210and the storage control bus 212.

In one embodiment, packets are written sequentially to the solid-statestorage 110. For example, packets are streamed to the storage writebuffers of a bank 214 a of storage elements 216 and when the buffers arefull, the packets are programmed to a designated virtual page. Packetsthen refill the storage write buffers and, when full, the packets arewritten to the next virtual page. The next virtual page may be in thesame bank 214 a or another bank (e.g. 214 b). This process continues,virtual page after virtual page, typically until a virtual erase blockis filled. In another embodiment, the streaming may continue acrossvirtual erase block boundaries with the process continuing, virtualerase block after virtual erase block.

In a read, modify, write operation, data packets associated with theobject are located and read in a read operation. Data segments of themodified object that have been modified are not written to the locationfrom which they are read. Instead, the modified data segments are againconverted to data packets and then written to the next availablelocation in the virtual page currently being written. The object indexentries for the respective data packets are modified to point to thepackets that contain the modified data segments. The entry or entries inthe object index for data packets associated with the same object thathave not been modified will include pointers to original location of theunmodified data packets. Thus, if the original object is maintained, forexample to maintain a previous version of the object, the originalobject will have pointers in the object index to all data packets asoriginally written. The new object will have pointers in the objectindex to some of the original data packets and pointers to the modifieddata packets in the virtual page that is currently being written.

In a copy operation, the object index includes an entry for the originalobject mapped to a number of packets stored in the solid-state storage110. When a copy is made, a new object is created and a new entry iscreated in the object index mapping the new object to the originalpackets. The new object is also written to the solid-state storage 110with its location mapped to the new entry in the object index. The newobject packets may be used to identify the packets within the originalobject that are referenced in case changes have been made in theoriginal object that have not been propagated to the copy and the objectindex is lost or corrupted.

Beneficially, sequentially writing packets facilitates a more even useof the solid-state storage 110 and allows the solid-storage devicecontroller 202 to monitor storage hot spots and level usage of thevarious virtual pages in the solid-state storage 110. Sequentiallywriting packets also facilitates a powerful, efficient garbagecollection system, which is described in detail below. One of skill inthe art will recognize other benefits of sequential storage of datapackets.

Solid-State Storage Device Controller

In various embodiments, the solid-state storage device controller 202also includes a data bus 204, a local bus 206, a buffer controller 208,buffers 0-N 222 a-n, a master controller 224, a direct memory access(“DMA”) controller 226, a memory controller 228, a dynamic memory array230, a static random memory array 232, a management controller 234, amanagement bus 236, a bridge 238 to a system bus 240, and miscellaneouslogic 242, which are described below. In other embodiments, the systembus 240 is coupled to one or more network interface cards (“NICs”) 244,some of which may include remote DMA (“RDMA”) controllers 246, one ormore central processing unit (“CPU”) 248, one or more external memorycontrollers 250 and associated external memory arrays 252, one or morestorage controllers 254, peer controllers 256, and application specificprocessors 258, which are described below. The components 244-258connected to the system bus 240 may be located in the computer 112 ormay be other devices.

Typically the solid-state storage controller(s) 104 communicate data tothe solid-state storage 110 over a storage I/O bus 210. In a typicalembodiment where the solid-state storage is arranged in banks 214 andeach bank 214 includes multiple storage elements 216, 218, 220 accessedin parallel, the storage I/O bus 210 is an array of busses, one for eachrow of storage elements 216, 218, 220 spanning the banks 214. As usedherein, the term “storage I/O bus” may refer to one storage I/O bus 210or an array of data independent busses 204. In a preferred embodiment,each storage I/O bus 210 accessing a row of storage elements (e.g. 216a, 218 a, 220 a) may include a logical-to-physical mapping for storagedivisions (e.g. erase blocks) accessed in a row of storage elements 216a, 218 a, 220 a. This mapping allows a logical address mapped to aphysical address of a storage division to be remapped to a differentstorage division if the first storage division fails, partially fails,is inaccessible, or has some other problem. Remapping is explainedfurther in relation to the remapping module 430 of FIGS. 4A and 4B.

Data may also be communicated to the solid-state storage controller(s)104 from a requesting device 155 through the system bus 240, bridge 238,local bus 206, buffer(s) 222, and finally over a data bus 204. The databus 204 typically is connected to one or more buffers 222 a-n controlledwith a buffer controller 208. The buffer controller 208 typicallycontrols transfer of data from the local bus 206 to the buffers 222 andthrough the data bus 204 to the pipeline input buffer 306 and outputbuffer 330. The buffer controller 208 typically controls how dataarriving from a requesting device 155 can be temporarily stored in abuffer 222 and then transferred onto a data bus 204, or vice versa, toaccount for different clock domains, to prevent data collisions, etc.The buffer controller 208 typically works in conjunction with the mastercontroller 224 to coordinate data flow. As data arrives, the data willarrive on the system bus 240, be transferred to the local bus 206through a bridge 238.

Typically the data is transferred from the local bus 206 to one or moredata buffers 222 as directed by the master controller 224 and the buffercontroller 208. The data then flows out of the buffer(s) 222 to the databus 204, through a solid-state controller 104, and on to the solid-statestorage 110 such as NAND flash or other storage media. In a preferredembodiment, data and associated out-of-band metadata (“object metadata”)arriving with the data is communicated using one or more data channelscomprising one or more solid-state storage controllers 104 a-104 n−1 andassociated solid-state storage 110 a-110 n−1 while at least one channel(solid-state storage controller 104 n, solid-state storage 110 n) isdedicated to in-band metadata, such as index information and othermetadata generated internally to the solid-state storage device 102.

The local bus 206 is typically a bidirectional bus or set of busses thatallows for communication of data and commands between devices internalto the solid-state storage device controller 202 and between devicesinternal to the solid-state storage device 102 and devices 244-258connected to the system bus 240. The bridge 238 facilitatescommunication between the local bus 206 and system bus 240. One of skillin the art will recognize other embodiments such as ring structures orswitched star configurations and functions of buses 240, 206, 204, 210and bridges 238.

The system bus 240 is typically a bus of a computer 112 or other devicein which the solid-state storage device 102 is installed or connected.In one embodiment, the system bus 240 may be a PCI-e bus, a SerialAdvanced Technology Attachment (“serial ATA”) bus, parallel ATA, or thelike. In another embodiment, the system bus 240 is an external bus suchas small computer system interface (“SCSI”), FireWire, Fiber Channel,USB, PCIe-AS, or the like. The solid-state storage device 102 may bepackaged to fit internally to a device or as an externally connecteddevice.

The solid-state storage device controller 202 includes a mastercontroller 224 that controls higher-level functions within thesolid-state storage device 102. The master controller 224, in variousembodiments, controls data flow by interpreting object requests andother requests, directs creation of indexes to map object identifiersassociated with data to physical locations of associated data,coordinating DMA requests, etc. Many of the functions described hereinare controlled wholly or in part by the master controller 224.

In one embodiment, the master controller 224 uses embeddedcontroller(s). In another embodiment, the master controller 224 useslocal memory such as a dynamic memory array 230 (dynamic random accessmemory “DRAM”), a static memory array 232 (static random access memory“SRAM”), etc. In one embodiment, the local memory is controlled usingthe master controller 224. In another embodiment, the master controller224 accesses the local memory via a memory controller 228. In anotherembodiment, the master controller 224 runs a Linux server and maysupport various common server interfaces, such as the World Wide Web,hyper-text markup language (“HTML”), etc. In another embodiment, themaster controller 224 uses a nano-processor. The master controller 224may be constructed using programmable or standard logic, or anycombination of controller types listed above. One skilled in the artwill recognize many embodiments for the master controller 224.

In one embodiment, where the storage controller 152/solid-state storagedevice controller 202 manages multiple data storage devices/solid-statestorage 110 a-n, the master controller 224 divides the work load amonginternal controllers, such as the solid-state storage controllers 104a-n. For example, the master controller 224 may divide an object to bewritten to the data storage devices (e.g. solid-state storage 110 a-n)so that a portion of the object is stored on each of the attached datastorage devices. This feature is a performance enhancement allowingquicker storage and access to an object. In one embodiment, the mastercontroller 224 is implemented using an FPGA. In another embodiment, thefirmware within the master controller 224 may be updated through themanagement bus 236, the system bus 240 over a network connected to a NIC244 or other device connected to the system bus 240.

In one embodiment, the master controller 224, which manages objects,emulates block storage such that a computer 112 or other deviceconnected to the storage device/solid-state storage device 102 views thestorage device/solid-state storage device 102 as a block storage deviceand sends data to specific physical addresses in the storagedevice/solid-state storage device 102. The master controller 224 thendivides up the blocks and stores the data blocks as it would objects.The master controller 224 then maps the blocks and physical address sentwith the block to the actual locations determined by the mastercontroller 224. The mapping is stored in the object index. Typically,for block emulation, a block device application program interface(“API”) is provided in a driver in the computer 112, client 114, orother device wishing to use the storage device/solid-state storagedevice 102 as a block storage device.

In another embodiment, the master controller 224 coordinates with NICcontrollers 244 and embedded RDMA controllers 246 to deliverjust-in-time RDMA transfers of data and command sets. NIC controller 244may be hidden behind a non-transparent port to enable the use of customdrivers. Also, a driver on a client 114 may have access to the computernetwork 116 through an I/O memory driver using a standard stack API andoperating in conjunction with NICs 244.

In one embodiment, the master controller 224 is also a redundant arrayof independent drive (“RAID”) controller. Where the data storagedevice/solid-state storage device 102 is networked with one or moreother data storage devices/solid-state storage devices 102, the mastercontroller 224 may be a RAID controller for single tier RAID, multi-tierRAID, progressive RAID, etc. The master controller 224 also allows someobjects to be stored in a RAID array and other objects to be storedwithout RAID. In another embodiment, the master controller 224 may be adistributed RAID controller element. In another embodiment, the mastercontroller 224 may comprise many RAID, distributed RAID, and otherfunctions as described elsewhere.

In one embodiment, the master controller 224 coordinates with single orredundant network managers (e.g. switches) to establish routing, tobalance bandwidth utilization, failover, etc. In another embodiment, themaster controller 224 coordinates with integrated application specificlogic (via local bus 206) and associated driver software. In anotherembodiment, the master controller 224 coordinates with attachedapplication specific processors 258 or logic (via the external systembus 240) and associated driver software. In another embodiment, themaster controller 224 coordinates with remote application specific logic(via the computer network 116) and associated driver software. Inanother embodiment, the master controller 224 coordinates with the localbus 206 or external bus attached hard disk drive (“HDD”) storagecontroller.

In one embodiment, the master controller 224 communicates with one ormore storage controllers 254 where the storage device/solid-statestorage device 102 may appear as a storage device connected through aSCSI bus, Internet SCSI (“iSC SI”), fiber channel, etc. Meanwhile thestorage device/solid-state storage device 102 may autonomously manageobjects and may appear as an object file system or distributed objectfile system. The master controller 224 may also be accessed by peercontrollers 256 and/or application specific processors 258.

In another embodiment, the master controller 224 coordinates with anautonomous integrated management controller to periodically validateFPGA code and/or controller software, validate FPGA code while running(reset) and/or validate controller software during power on (reset),support external reset requests, support reset requests due to watchdogtimeouts, and support voltage, current, power, temperature, and otherenvironmental measurements and setting of threshold interrupts. Inanother embodiment, the master controller 224 manages garbage collectionto free erase blocks for reuse. In another embodiment, the mastercontroller 224 manages wear leveling. In another embodiment, the mastercontroller 224 allows the data storage device/solid-state storage device102 to be partitioned into multiple virtual devices and allowspartition-based media encryption. In yet another embodiment, the mastercontroller 224 supports a solid-state storage controller 104 withadvanced, multi-bit ECC correction. One of skill in the art willrecognize other features and functions of a master controller 224 in astorage controller 152, or more specifically in a solid-state storagedevice 102.

In one embodiment, the solid-state storage device controller 202includes a memory controller 228 which controls a dynamic random memoryarray 230 and/or a static random memory array 232. As stated above, thememory controller 228 may be independent or integrated with the mastercontroller 224. The memory controller 228 typically controls volatilememory of some type, such as DRAM (dynamic random memory array 230) andSRAM (static random memory array 232). In other examples, the memorycontroller 228 also controls other memory types such as electricallyerasable programmable read only memory (“EEPROM”), etc. In otherembodiments, the memory controller 228 controls two or more memory typesand the memory controller 228 may include more than one controller.Typically, the memory controller 228 controls as much SRAM 232 as isfeasible and by DRAM 230 to supplement the SRAM 232.

In one embodiment, the object index is stored in memory 230, 232 andthen periodically off-loaded to a channel of the solid-state storage 110n or other non-volatile memory. One of skill in the art will recognizeother uses and configurations of the memory controller 228, dynamicmemory array 230, and static memory array 232.

In one embodiment, the solid-state storage device controller 202includes a DMA controller 226 that controls DMA operations between thestorage device/solid-state storage device 102 and one or more externalmemory controllers 250 and associated external memory arrays 252 andCPUs 248. Note that the external memory controllers 250 and externalmemory arrays 252 are called external because they are external to thestorage device/solid-state storage device 102. In addition the DMAcontroller 226 may also control RDMA operations with requesting devicesthrough a NIC 244 and associated RDMA controller 246. DMA and RDMA areexplained in more detail below.

In one embodiment, the solid-state storage device controller 202includes a management controller 234 connected to a management bus 236.Typically the management controller 234 manages environmental metricsand status of the storage device/solid-state storage device 102. Themanagement controller 234 may monitor device temperature, fan speed,power supply settings, etc. over the management bus 236. The managementcontroller 234 may support the reading and programming of erasableprogrammable read only memory (“EEPROM”) for storage of FPGA code andcontroller software. Typically the management bus 236 is connected tothe various components within the storage device/solid-state storagedevice 102. The management controller 234 may communicate alerts,interrupts, etc. over the local bus 206 or may include a separateconnection to a system bus 240 or other bus. In one embodiment themanagement bus 236 is an Inter-Integrated Circuit (“I²C”) bus. One ofskill in the art will recognize other related functions and uses of amanagement controller 234 connected to components of the storagedevice/solid-state storage device 102 by a management bus 236.

In one embodiment, the solid-state storage device controller 202includes miscellaneous logic 242 that may be customized for a specificapplication. Typically where the solid-state device controller 202 ormaster controller 224 is/are configured using a FPGA or otherconfigurable controller, custom logic may be included based on aparticular application, customer requirement, storage requirement, etc.

Data Pipeline

FIG. 3 is a schematic block diagram illustrating one embodiment 300 of asolid-state storage controller 104 with a write data pipeline 106 and aread data pipeline 108 in a solid-state storage device 102 in accordancewith the present invention. The embodiment 300 includes a data bus 204,a local bus 206, and buffer control 208, which are substantially similarto those described in relation to the solid-state storage devicecontroller 202 of FIG. 2. The write data pipeline 106 includes apacketizer 302 and an error-correcting code (“ECC”) generator 304. Inother embodiments, the write data pipeline 106 includes an input buffer306, a write synchronization buffer 308, a write program module 310, acompression module 312, an encryption module 314, a garbage collectorbypass 316 (with a portion within the read data pipeline 108), a mediaencryption module 318, and a write buffer 320. The read data pipeline108 includes a read synchronization buffer 328, an ECC correction module322, a depacketizer 324, an alignment module 326, and an output buffer330. In other embodiments, the read data pipeline 108 may include amedia decryption module 332, a portion of the garbage collector bypass316, a decryption module 334, a decompression module 336, and a readprogram module 338. The solid-state storage controller 104 may alsoinclude control and status registers 340 and control queues 342, a bankinterleave controller 344, a synchronization buffer 346, a storage buscontroller 348, and a multiplexer (“MUX”) 350. The components of thesolid-state controller 104 and associated write data pipeline 106 andread data pipeline 108 are described below. In other embodiments,synchronous solid-state storage 110 may be used and synchronizationbuffers 308 328 may be eliminated.

Write Data Pipeline

The write data pipeline 106 includes a packetizer 302 that receives adata or metadata segment to be written to the solid-state storage,either directly or indirectly through another write data pipeline 106stage, and creates one or more packets sized for the solid-state storage110. The data or metadata segment is typically part of an object, butmay also include an entire object. In another embodiment, the datasegment is part of a block of data, but may also include an entire blockof data. Typically, an object is received from a computer 112, client114, or other computer or device and is transmitted to the solid-statestorage device 102 in data segments streamed to the solid-state storagedevice 102 or computer 112. A data segment may also be known by anothername, such as data parcel, but as referenced herein includes all or aportion of an object or data block.

Each object is stored as one or more packets. Each object may have oneor more container packets. Each packet contains a header. The header mayinclude a header type field. Type fields may include data, objectattribute, metadata, data segment delimiters (multi-packet), objectstructures, object linkages, and the like. The header may also includeinformation regarding the size of the packet, such as the number ofbytes of data included in the packet. The length of the packet may beestablished by the packet type. The header may include information thatestablishes the relationship of the packet to the object. An examplemight be the use of an offset in a data packet header to identify thelocation of the data segment within the object. One of skill in the artwill recognize other information that may be included in a header addedto data by a packetizer 302 and other information that may be added to adata packet.

Each packet includes a header and possibly data from the data ormetadata segment. The header of each packet includes pertinentinformation to relate the packet to the object to which the packetbelongs. For example, the header may include an object identifier andoffset that indicates the data segment, object, or data block from whichthe data packet was formed. The header may also include a logicaladdress used by the storage bus controller 348 to store the packet. Theheader may also include information regarding the size of the packet,such as the number of bytes included in the packet. The header may alsoinclude a sequence number that identifies where the data segment belongswith respect to other packets within the object when reconstructing thedata segment or object. The header may include a header type field. Typefields may include data, object attributes, metadata, data segmentdelimiters (multi-packet), object structures, object linkages, and thelike. One of skill in the art will recognize other information that maybe included in a header added to data or metadata by a packetizer 302and other information that may be added to a packet.

The write data pipeline 106 includes an ECC generator 304 that generatesone or more error-correcting codes (“ECC”) for the one or more packetsreceived from the packetizer 302. The ECC generator 304 typically usesan error correcting algorithm to generate ECC which is stored with thepacket. The ECC stored with the packet is typically used to detect andcorrect errors introduced into the data through transmission andstorage.

In one embodiment, packets are streamed into the ECC generator 304 asun-encoded blocks of length N. A syndrome of length S is calculated,appended and output as an encoded block of length N+S. The value of Nand S are dependent upon the characteristics of the algorithm which isselected to achieve specific performance, efficiency, and robustnessmetrics. In the preferred embodiment, there is no fixed relationshipbetween the ECC blocks and the packets; the packet may comprise morethan one ECC block; the ECC block may comprise more than one packet; anda first packet may end anywhere within the ECC block and a second packetmay begin after the end of the first packet within the same ECC block.In the preferred embodiment, ECC algorithms are not dynamicallymodified. In a preferred embodiment, the ECC stored with the datapackets is robust enough to correct errors in more than two bits.

Beneficially, using a robust ECC algorithm allowing more than single bitcorrection or even double bit correction allows the life of thesolid-state storage 110 to be extended. For example, if flash memory isused as the storage medium in the solid-state storage 110, the flashmemory may be written approximately 100,000 times without error pererase cycle. This usage limit may be extended using a robust ECCalgorithm. Having the ECC generator 304 and corresponding ECC correctionmodule 322 onboard the solid-state storage device 102, the solid-statestorage device 102 can internally correct errors and has a longer usefullife than if a less robust ECC algorithm is used, such as single bitcorrection. However, in other embodiments the ECC generator 304 may usea less robust algorithm and may correct single-bit or double-bit errors.In another embodiment, the solid-state storage device 110 may compriseless reliable storage such as multi-level cell (“MLC”) flash in order toincrease capacity, which storage may not be sufficiently reliablewithout more robust ECC algorithms.

In one embodiment, the write pipeline 106 includes an input buffer 306that receives a data segment to be written to the solid-state storage110 and stores the incoming data segments until the next stage of thewrite data pipeline 106, such as the packetizer 302 (or other stage fora more complex write data pipeline 106) is ready to process the nextdata segment. The input buffer 306 typically allows for discrepanciesbetween the rate data segments are received and processed by the writedata pipeline 106 using an appropriately sized data buffer. The inputbuffer 306 also allows the data bus 204 to transfer data to the writedata pipeline 106 at rates greater than can be sustained by the writedata pipeline 106 in order to improve efficiency of operation of thedata bus 204. Typically when the write data pipeline 106 does notinclude an input buffer 306, a buffering function is performedelsewhere, such as in the solid-state storage device 102 but outside thewrite data pipeline 106, in the computer 112, such as within a networkinterface card (“NIC”), or at another device, for example when usingremote direct memory access (“RDMA”).

In another embodiment, the write data pipeline 106 also includes a writesynchronization buffer 308 that buffers packets received from the ECCgenerator 304 prior to writing the packets to the solid-state storage110. The write synch buffer 308 is located at a boundary between a localclock domain and a solid-state storage clock domain and providesbuffering to account for the clock domain differences. In otherembodiments, synchronous solid-state storage 110 may be used andsynchronization buffers 308 328 may be eliminated.

In one embodiment, the write data pipeline 106 also includes a mediaencryption module 318 that receives the one or more packets from thepacketizer 302, either directly or indirectly, and encrypts the one ormore packets using an encryption key unique to the solid-state storagedevice 102 prior to sending the packets to the ECC generator 304.Typically, the entire packet is encrypted, including the headers. Inanother embodiment, headers are not encrypted. In this document,encryption key is understood to mean a secret encryption key that ismanaged externally from an embodiment that integrates the solid-statestorage 110 and where the embodiment requires encryption protection. Themedia encryption module 318 and corresponding media decryption module332 provide a level of security for data stored in the solid-statestorage 110. For example, where data is encrypted with the mediaencryption module 318, if the solid-state storage 110 is connected to adifferent solid-state storage controller 104, solid-state storage device102, or computer 112, the contents of the solid-state storage 110typically could not be read without use of the same encryption key usedduring the write of the data to the solid-state storage 110 withoutsignificant effort.

In a typical embodiment, the solid-state storage device 102 does notstore the encryption key in non-volatile storage and allows no externalaccess to the encryption key. The encryption key is provided to thesolid-state storage controller 104 during initialization. The solid-satestorage device 102 may use and store a non-secret cryptographic noncethat is used in conjunction with an encryption key. A different noncemay be stored with every packet. Data segments may be split betweenmultiple packets with unique nonces for the purpose of improvingprotection by the encryption algorithm. The encryption key may bereceived from a client 114, a computer 112, key manager, or other devicethat manages the encryption key to be used by the solid-state storagecontroller 104. In another embodiment, the solid-state storage 110 mayhave two or more partitions and the solid-state storage controller 104behaves as though it were two or more solid-state storage controllers104, each operating on a single partition within the solid-state storage110. In this embodiment, a unique media encryption key may be used witheach partition.

In another embodiment, the write data pipeline 106 also includes anencryption module 314 that encrypts a data or metadata segment receivedfrom the input buffer 306, either directly or indirectly, prior sendingthe data segment to the packetizer 302, the data segment encrypted usingan encryption key received in conjunction with the data segment. Theencryption module 314 differs from the media encryption module 318 inthat the encryption keys used by the encryption module 314 to encryptdata may not be common to all data stored within the solid-state storagedevice 102 but may vary on an object basis and received in conjunctionwith receiving data segments as described below. For example, anencryption key for a data segment to be encrypted by the encryptionmodule 314 may be received with the data segment or may be received aspart of a command to write an object to which the data segment belongs.The solid-sate storage device 102 may use and store a non-secretcryptographic nonce in each object packet that is used in conjunctionwith the encryption key. A different nonce may be stored with everypacket. Data segments may be split between multiple packets with uniquenonces for the purpose of improving protection by the encryptionalgorithm. In one embodiment, the nonce used by the media encryptionmodule 318 is the same as that used by the encryption module 314.

The encryption key may be received from a client 114, a computer 112,key manager, or other device that holds the encryption key to be used toencrypt the data segment. In one embodiment, encryption keys aretransferred to the solid-state storage controller 104 from one of asolid-state storage device 102, computer 112, client 114, or otherexternal agent which has the ability to execute industry standardmethods to securely transfer and protect private and public keys.

In one embodiment, the encryption module 314 encrypts a first packetwith a first encryption key received in conjunction with the packet andencrypts a second packet with a second encryption key received inconjunction with the second packet. In another embodiment, theencryption module 314 encrypts a first packet with a first encryptionkey received in conjunction with the packet and passes a second datapacket on to the next stage without encryption. Beneficially, theencryption module 314 included in the write data pipeline 106 of thesolid-state storage device 102 allows object-by-object orsegment-by-segment data encryption without a single file system or otherexternal system to keep track of the different encryption keys used tostore corresponding objects or data segments. Each requesting device 155or related key manager independently manages encryption keys used toencrypt only the objects or data segments sent by the requesting device155.

In another embodiment, the write data pipeline 106 includes acompression module 312 that compresses the data for metadata segmentprior to sending the data segment to the packetizer 302. The compressionmodule 312 typically compresses a data or metadata segment using acompression routine known to those of skill in the art to reduce thestorage size of the segment. For example, if a data segment includes astring of 512 zeros, the compression module 312 may replace the 512zeros with code or token indicating the 512 zeros where the code is muchmore compact than the space taken by the 512 zeros.

In one embodiment, the compression module 312 compresses a first segmentwith a first compression routine and passes along a second segmentwithout compression. In another embodiment, the compression module 312compresses a first segment with a first compression routine andcompresses the second segment with a second compression routine. Havingthis flexibility within the solid-state storage device 102 is beneficialso that clients 114 or other devices writing data to the solid-statestorage device 102 may each specify a compression routine or so that onecan specify a compression routine while another specifies nocompression. Selection of compression routines may also be selectedaccording to default settings on a per object type or object classbasis. For example, a first object of a specific object may be able tooverride default compression routine settings and a second object of thesame object class and object type may use the default compressionroutine and a third object of the same object class and object type mayuse no compression.

In one embodiment, the write data pipeline 106 includes a garbagecollector bypass 316 that receives data segments from the read datapipeline 108 as part of a data bypass in a garbage collection system. Agarbage collection system typically marks packets that are no longervalid, typically because the packet is marked for deletion or has beenmodified and the modified data is stored in a different location. Atsome point, the garbage collection system determines that a particularsection of storage may be recovered. This determination may be due to alack of available storage capacity, the percentage of data marked asinvalid reaching a threshold, a consolidation of valid data, an errordetection rate for that section of storage reaching a threshold, orimproving performance based on data distribution, etc. Numerous factorsmay be considered by a garbage collection algorithm to determine when asection of storage is to be recovered.

Once a section of storage has been marked for recovery, valid packets inthe section typically must be relocated. The garbage collector bypass316 allows packets to be read into the read data pipeline 108 and thentransferred directly to the write data pipeline 106 without being routedout of the solid-state storage controller 104. In a preferredembodiment, the garbage collector bypass 316 is part of an autonomousgarbage collector system that operates within the solid-state storagedevice 102. This allows the solid-state storage device 102 to managedata so that data is systematically spread throughout the solid-statestorage 110 to improve performance, data reliability and to avoidoveruse and underuse of any one location or area of the solid-statestorage 110 and to lengthen the useful life of the solid-state storage110.

The garbage collector bypass 316 coordinates insertion of segments intothe write data pipeline 106 with other segments being written by clients114 or other devices. In the depicted embodiment, the garbage collectorbypass 316 is before the packetizer 302 in the write data pipeline 106and after the depacketizer 324 in the read data pipeline 108, but mayalso be located elsewhere in the read and write data pipelines 106, 108.The garbage collector bypass 316 may be used during a flush of the writepipeline 106 to fill the remainder of the virtual page in order toimprove the efficiency of storage within the Solid-State Storage 110 andthereby reduce the frequency of garbage collection.

In one embodiment, the write data pipeline 106 includes a write buffer320 that buffers data for efficient write operations. Typically, thewrite buffer 320 includes enough capacity for packets to fill at leastone virtual page in the solid-state storage 110. This allows a writeoperation to send an entire page of data to the solid-state storage 110without interruption. By sizing the write buffer 320 of the write datapipeline 106 and buffers within the read data pipeline 108 to be thesame capacity or larger than a storage write buffer within thesolid-state storage 110, writing and reading data is more efficientsince a single write command may be crafted to send a full virtual pageof data to the solid-state storage 110 instead of multiple commands.

While the write buffer 320 is being filled, the solid-state storage 110may be used for other read operations. This is advantageous becauseother solid-state devices with a smaller write buffer or no write buffermay tie up the solid-state storage when data is written to a storagewrite buffer and data flowing into the storage write buffer stalls. Readoperations will be blocked until the entire storage write buffer isfilled and programmed. Another approach for systems without a writebuffer or a small write buffer is to flush the storage write buffer thatis not full in order to enable reads. Again this is inefficient becausemultiple write/program cycles are required to fill a page.

For depicted embodiment with a write buffer 320 sized larger than avirtual page, a single write command, which includes numeroussubcommands, can then be followed by a single program command totransfer the page of data from the storage write buffer in eachsolid-state storage element 216, 218, 220 to the designated page withineach solid-state storage element 216, 218, 220. This technique has thebenefits of eliminating partial page programming, which is known toreduce data reliability and durability and freeing up the destinationbank for reads and other commands while the buffer fills.

In one embodiment, the write buffer 320 is a ping-pong buffer where oneside of the buffer is filled and then designated for transfer at anappropriate time while the other side of the ping-pong buffer is beingfilled. In another embodiment, the write buffer 320 includes a first-infirst-out (“FIFO”) register with a capacity of more than a virtual pageof data segments. One of skill in the art will recognize other writebuffer 320 configurations that allow a virtual page of data to be storedprior to writing the data to the solid-state storage 110.

In another embodiment, the write buffer 320 is sized smaller than avirtual page so that less than a page of information could be written toa storage write buffer in the solid-state storage 110. In theembodiment, to prevent a stall in the write data pipeline 106 fromholding up read operations, data is queued using the garbage collectionsystem that needs to be moved from one location to another as part ofthe garbage collection process. In case of a data stall in the writedata pipeline 106, the data can be fed through the garbage collectorbypass 316 to the write buffer 320 and then on to the storage writebuffer in the solid-state storage 110 to fill the pages of a virtualpage prior to programming the data. In this way a data stall in thewrite data pipeline 106 would not stall reading from the solid-statestorage device 102.

In another embodiment, the write data pipeline 106 includes a writeprogram module 310 with one or more user-definable functions within thewrite data pipeline 106. The write program module 310 allows a user tocustomize the write data pipeline 106. A user may customize the writedata pipeline 106 based on a particular data requirement or application.Where the solid-state storage controller 104 is an FPGA, the user mayprogram the write data pipeline 106 with custom commands and functionsrelatively easily. A user may also use the write program module 310 toinclude custom functions with an ASIC, however, customizing an ASIC maybe more difficult than with an FPGA. The write program module 310 mayinclude buffers and bypass mechanisms to allow a first data segment toexecute in the write program module 310 while a second data segment maycontinue through the write data pipeline 106. In another embodiment, thewrite program module 310 may include a processor core that can beprogrammed through software.

Note that the write program module 310 is shown between the input buffer306 and the compression module 312, however, the write program module310 could be anywhere in the write data pipeline 106 and may bedistributed among the various stages 302-320. In addition, there may bemultiple write program modules 310 distributed among the various states302-320 that are programmed and operate independently. In addition, theorder of the stages 302-320 may be altered. One of skill in the art willrecognize workable alterations to the order of the stages 302-320 basedon particular user requirements.

Read Data Pipeline

The read data pipeline 108 includes an ECC correction module 322 thatdetermines if a data error exists in the ECC blocks of a requestedpacket received from the solid-state storage 110 by using the ECC storedwith each ECC block of the requested packet. The ECC correction module322 then corrects any errors in the requested packet if any error existsand the errors are correctable using the ECC. For example, if the ECCcan detect an error in six bits but can only correct three bit errors,the ECC correction module 322 corrects ECC blocks of the requestedpacket with up to three bits in error. The ECC correction module 322corrects the bits in error by changing the bits in error to the correctone or zero state so that the requested data packet is identical to whenit was written to the solid-state storage 110 and the ECC was generatedfor the packet.

If the ECC correction module 322 determines that the requested packetscontains more bits in error than the ECC can correct, the ECC correctionmodule 322 cannot correct the errors in the corrupted ECC blocks of therequested packet and sends an interrupt. In one embodiment, the ECCcorrection module 322 sends an interrupt with a message indicating thatthe requested packet is in error. The message may include informationthat the ECC correction module 322 cannot correct the errors or theinability of the ECC correction module 322 to correct the errors may beimplied. In another embodiment, the ECC correction module 322 sends thecorrupted ECC blocks of the requested packet with the interrupt and/orthe message.

In the preferred embodiment, a corrupted ECC block or portion of acorrupted ECC block of the requested packet that cannot be corrected bythe ECC correction module 322 is read by the master controller 224,corrected, and returned to the ECC correction module 322 for furtherprocessing by the read data pipeline 108. In one embodiment, a corruptedECC block or portion of a corrupted ECC block of the requested packet issent to the device requesting the data. The requesting device 155 maycorrect the ECC block or replace the data using another copy, such as abackup or mirror copy, and then may use the replacement data of therequested data packet or return it to the read data pipeline 108. Therequesting device 155 may use header information in the requested packetin error to identify data required to replace the corrupted requestedpacket or to replace the object to which the packet belongs. In anotherpreferred embodiment, the solid-state storage controller 104 stores datausing some type of RAID and is able to recover the corrupted data. Inanother embodiment, the ECC correction module 322 sends and interruptand/or message and the receiving device fails the read operationassociated with the requested data packet. One of skill in the art willrecognize other options and actions to be taken as a result of the ECCcorrection module 322 determining that one or more ECC blocks of therequested packet are corrupted and that the ECC correction module 322cannot correct the errors.

The read data pipeline 108 includes a depacketizer 324 that receives ECCblocks of the requested packet from the ECC correction module 322,directly or indirectly, and checks and removes one or more packetheaders. The depacketizer 324 may validate the packet headers bychecking packet identifiers, data length, data location, etc. within theheaders. In one embodiment, the header includes a hash code that can beused to validate that the packet delivered to the read data pipeline 108is the requested packet. The depacketizer 324 also removes the headersfrom the requested packet added by the packetizer 302. The depacketizer324 may directed to not operate on certain packets but pass theseforward without modification. An example might be a container label thatis requested during the course of a rebuild process where the headerinformation is required by the object index reconstruction module 272.Further examples include the transfer of packets of various typesdestined for use within the solid-state storage device 102. In anotherembodiment, the depacketizer 324 operation may be packet type dependent.

The read data pipeline 108 includes an alignment module 326 thatreceives data from the depacketizer 324 and removes unwanted data. Inone embodiment, a read command sent to the solid-state storage 110retrieves a packet of data. A device requesting the data may not requireall data within the retrieved packet and the alignment module 326removes the unwanted data. If all data within a retrieved page isrequested data, the alignment module 326 does not remove any data.

The alignment module 326 re-formats the data as data segments of anobject in a form compatible with a device requesting the data segmentprior to forwarding the data segment to the next stage. Typically, asdata is processed by the read data pipeline 108, the size of datasegments or packets changes at various stages. The alignment module 326uses received data to format the data into data segments suitable to besent to the requesting device 155 and joined to form a response. Forexample, data from a portion of a first data packet may be combined withdata from a portion of a second data packet. If a data segment is largerthan a data requested by the requesting device 155, the alignment module326 may discard the unwanted data.

In one embodiment, the read data pipeline 108 includes a readsynchronization buffer 328 that buffers one or more requested packetsread from the solid-state storage 110 prior to processing by the readdata pipeline 108. The read synchronization buffer 328 is at theboundary between the solid-state storage clock domain and the local busclock domain and provides buffering to account for the clock domaindifferences.

In another embodiment, the read data pipeline 108 includes an outputbuffer 330 that receives requested packets from the alignment module 326and stores the packets prior to transmission to the requesting device155. The output buffer 330 accounts for differences between when datasegments are received from stages of the read data pipeline 108 and whenthe data segments are transmitted to other parts of the solid-statestorage controller 104 or to the requesting device 155. The outputbuffer 330 also allows the data bus 204 to receive data from the readdata pipeline 108 at rates greater than can be sustained by the readdata pipeline 108 in order to improve efficiency of operation of thedata bus 204.

In one embodiment, the read data pipeline 108 includes a mediadecryption module 332 that receives one or more encrypted requestedpackets from the ECC correction module 322 and decrypts the one or morerequested packets using the encryption key unique to the solid-statestorage device 102 prior to sending the one or more requested packets tothe depacketizer 324. Typically the encryption key used to decrypt databy the media decryption module 332 is identical to the encryption keyused by the media encryption module 318. In another embodiment, thesolid-state storage 110 may have two or more partitions and thesolid-state storage controller 104 behaves as though it were two or moresolid-state storage controllers 104 each operating on a single partitionwithin the solid-state storage 110. In this embodiment, a unique mediaencryption key may be used with each partition.

In another embodiment, the read data pipeline 108 includes a decryptionmodule 334 that decrypts a data segment formatted by the depacketizer324 prior to sending the data segment to the output buffer 330. The datasegment decrypted using an encryption key received in conjunction withthe read request that initiates retrieval of the requested packetreceived by the read synchronization buffer 328. The decryption module334 may decrypt a first packet with an encryption key received inconjunction with the read request for the first packet and then maydecrypt a second packet with a different encryption key or may pass thesecond packet on to the next stage of the read data pipeline 108 withoutdecryption. Typically, the decryption module 334 uses a differentencryption key to decrypt a data segment than the media decryptionmodule 332 uses to decrypt requested packets. When the packet was storedwith a non-secret cryptographic nonce, the nonce is used in conjunctionwith an encryption key to decrypt the data packet. The encryption keymay be received from a client 114, a computer 112, key manager, or otherdevice that manages the encryption key to be used by the solid-statestorage controller 104.

In another embodiment, the read data pipeline 108 includes adecompression module 336 that decompresses a data segment formatted bythe depacketizer 324. In the preferred embodiment, the decompressionmodule 336 uses compression information stored in one or both of thepacket header and the container label to select a complementary routineto that used to compress the data by the compression module 312. Inanother embodiment, the decompression routine used by the decompressionmodule 336 is dictated by the device requesting the data segment beingdecompressed. In another embodiment, the decompression module 336selects a decompression routine according to default settings on a perobject type or object class basis. A first packet of a first object maybe able to override a default decompression routine and a second packetof a second object of the same object class and object type may use thedefault decompression routine and a third packet of a third object ofthe same object class and object type may use no decompression.

In another embodiment, the read data pipeline 108 includes a readprogram module 338 that includes one or more user-definable functionswithin the read data pipeline 108. The read program module 338 hassimilar characteristics to the write program module 310 and allows auser to provide custom functions to the read data pipeline 108. The readprogram module 338 may be located as shown in FIG. 3, may be located inanother position within the read data pipeline 108, or may includemultiple parts in multiple locations within the read data pipeline 108.Additionally, there may be multiple read program modules 338 withinmultiple locations within the read data pipeline 108 that operateindependently. One of skill in the art will recognize other forms of aread program module 338 within a read data pipeline 108. As with thewrite data pipeline 106, the stages of the read data pipeline 108 may berearranged and one of skill in the art will recognize other orders ofstages within the read data pipeline 108.

The solid-state storage controller 104 includes control and statusregisters 340 and corresponding control queues 342. The control andstatus registers 340 and control queues 342 facilitate control andsequencing commands and subcommands associated with data processed inthe write and read data pipelines 106, 108. For example, a data segmentin the packetizer 302 may have one or more corresponding controlcommands or instructions in a control queue 342 associated with the ECCgenerator 304. As the data segment is packetized, some of theinstructions or commands may be executed within the packetizer 302.Other commands or instructions may be passed to the next control queue342 through the control and status registers 340 as the newly formeddata packet created from the data segment is passed to the next stage.

Commands or instructions may be simultaneously loaded into the controlqueues 342 for a packet being forwarded to the write data pipeline 106with each pipeline stage pulling the appropriate command or instructionas the respective packet is executed by that stage. Similarly, commandsor instructions may be simultaneously loaded into the control queues 342for a packet being requested from the read data pipeline 108 with eachpipeline stage pulling the appropriate command or instruction as therespective packet is executed by that stage. One of skill in the artwill recognize other features and functions of control and statusregisters 340 and control queues 342.

The solid-state storage controller 104 and or solid-state storage device102 may also include a bank interleave controller 344, a synchronizationbuffer 346, a storage bus controller 348, and a multiplexer (“MUX”) 350,which are described in relation to FIGS. 4A and 4B.

Bank Interleave

FIG. 4A is a schematic block diagram illustrating one embodiment 400 ofa bank interleave controller 344 in the solid-state storage controller104 in accordance with the present invention. The bank interleavecontroller 344 is connected to the control and status registers 340 andto the storage I/O bus 210 and storage control bus 212 through the MUX350, storage bus controller 348, and synchronization buffer 346, whichare described below. The bank interleave controller 344 includes a readagent 402, a write agent 404, an erase agent 406, a management agent408, read queues 410 a-n, write queues 412 a-n, erase queues 414 a-n,and management queues 416 a-n for the banks 214 in the solid-statestorage 110, bank controllers 418 a-n, a bus arbiter 420, and a statusMUX 422, which are described below. The storage bus controller 348includes a mapping module 424 with a remapping module 430, a statuscapture module 426, and a NAND bus controller 428, which are describedbelow.

The bank interleave controller 344 directs one or more commands to twoor more queues in the bank interleave controller 344 and coordinatesamong the banks 214 of the solid-state storage 110 execution of thecommands stored in the queues, such that a command of a first typeexecutes on one bank 214 a while a command of a second type executes ona second bank 214 b. The one or more commands are separated by commandtype into the queues. Each bank 214 of the solid-state storage 110 has acorresponding set of queues within the bank interleave controller 344and each set of queues includes a queue for each command type.

The bank interleave controller 344 coordinates among the banks 214 ofthe solid-state storage 110 execution of the commands stored in thequeues. For example, a command of a first type executes on one bank 214a while a command of a second type executes on a second bank 214 b.Typically the command types and queue types include read and writecommands and queues 410, 412, but may also include other commands andqueues that are storage media specific. For example, in the embodimentdepicted in FIG. 4A, erase and management queues 414, 416 are includedand would be appropriate for flash memory, NRAM, MRAM, DRAM, PRAM, etc.

For other types of solid-state storage 110, other types of commands andcorresponding queues may be included without straying from the scope ofthe invention. The flexible nature of an FPGA solid-state storagecontroller 104 allows flexibility in storage media. If flash memory werechanged to another solid-state storage type, the bank interleavecontroller 344, storage bus controller 348, and MUX 350 could be alteredto accommodate the media type without significantly affecting the datapipelines 106, 108 and other solid-state storage controller 104functions.

In the embodiment depicted in FIG. 4A, the bank interleave controller344 includes, for each bank 214, a read queue 410 for reading data fromthe solid-state storage 110, a write queue 412 for write commands to thesolid-state storage 110, an erase queue 414 for erasing an erase blockin the solid-state storage, an a management queue 416 for managementcommands. The bank interleave controller 344 also includes correspondingread, write, erase, and management agents 402, 404, 406, 408. In anotherembodiment, the control and status registers 340 and control queues 342or similar components queue commands for data sent to the banks 214 ofthe solid-state storage 110 without a bank interleave controller 344.

The agents 402, 404, 406, 408, in one embodiment, direct commands of theappropriate type destined for a particular bank 214 a to the correctqueue for the bank 214 a. For example, the read agent 402 may receive aread command for bank-1 214 b and directs the read command to the bank-1read queue 410 b. The write agent 404 may receive a write command towrite data to a location in bank-0 214 a of the solid-state storage 110and will then send the write command to the bank-0 write queue 412 a.Similarly, the erase agent 406 may receive an erase command to erase anerase block in bank-1 214 b and will then pass the erase command to thebank-1 erase queue 414 b. The management agent 408 typically receivesmanagement commands, status requests, and the like, such as a resetcommand or a request to read a configuration register of a bank 214,such as bank-0 214 a. The management agent 408 sends the managementcommand to the bank-0 management queue 416 a.

The agents 402, 404, 406, 408 typically also monitor status of thequeues 410, 412, 414, 416 and send status, interrupt, or other messageswhen the queues 410, 412, 414, 416 are full, nearly full,non-functional, etc. In one embodiment, the agents 402, 404, 406, 408receive commands and generate corresponding sub-commands. In oneembodiment, the agents 402, 404, 406, 408 receive commands through thecontrol & status registers 340 and generate corresponding sub-commandswhich are forwarded to the queues 410, 412, 414, 416. One of skill inthe art will recognize other functions of the agents 402, 404, 406, 408.

The queues 410, 412, 414, 416 typically receive commands and store thecommands until required to be sent to the solid-state storage banks 214.In a typical embodiment, the queues 410, 412, 414, 416 are first-in,first-out (“FIFO”) registers or a similar component that operates as aFIFO. In another embodiment, the queues 410, 412, 414, 416 storecommands in an order that matches data, order of importance, or othercriteria.

The bank controllers 418 typically receive commands from the queues 410,412, 414, 416 and generate appropriate subcommands. For example, thebank-0 write queue 412 a may receive a command to write a page of datapackets to bank-0 214 a. The bank-0 controller 418 a may receive thewrite command at an appropriate time and may generate one or more writesubcommands for each data packet stored in the write buffer 320 to bewritten to the page in bank-0 214 a. For example, bank-0 controller 418a may generate commands to validate the status of bank 0 214 a and thesolid-state storage array 216, select the appropriate location forwriting one or more data packets, clear the input buffers within thesolid-state storage memory array 216, transfer the one or more datapackets to the input buffers, program the input buffers into theselected location, verify that the data was correctly programmed, and ifprogram failures occur do one or more of interrupting the mastercontroller 224, retrying the write to the same physical location, andretrying the write to a different physical location. Additionally, inconjunction with example write command, the storage bus controller 348will cause the one or more commands to multiplied to each of the each ofthe storage I/O buses 210 a-n with the logical address of the commandmapped to a first physical addresses for storage I/O bus 210 a, andmapped to a second physical address for storage I/O bus 210 b, and soforth as further described below.

Typically, bus arbiter 420 selects from among the bank controllers 418and pulls subcommands from output queues within the bank controllers 418and forwards these to the Storage Bus Controller 348 in a sequence thatoptimizes the performance of the banks 214. In another embodiment, thebus arbiter 420 may respond to a high level interrupt and modify thenormal selection criteria. In another embodiment, the master controller224 can control the bus arbiter 420 through the control and statusregisters 340. One of skill in the art will recognize other means bywhich the bus arbiter 420 may control and interleave the sequence ofcommands from the bank controllers 418 to the solid-state storage 110.

The bus arbiter 420 typically coordinates selection of appropriatecommands, and corresponding data when required for the command type,from the bank controllers 418 and sends the commands and data to thestorage bus controller 348. The bus arbiter 420 typically also sendscommands to the storage control bus 212 to select the appropriate bank214. For the case of flash memory or other solid-state storage 110 withan asynchronous, bi-directional serial storage I/O bus 210, only onecommand (control information) or set of data can be transmitted at atime. For example, when write commands or data are being transmitted tothe solid-state storage 110 on the storage I/O bus 210, read commands,data being read, erase commands, management commands, or other statuscommands cannot be transmitted on the storage I/O bus 210. For example,when data is being read from the storage I/O bus 210, data cannot bewritten to the solid-state storage 110.

For example, during a write operation on bank-0 the bus arbiter 420selects the bank-0 controller 418 a which may have a write command or aseries of write sub-commands on the top of its queue which cause thestorage bus controller 348 to execute the following sequence. The busarbiter 420 forwards the write command to the storage bus controller348, which sets up a write command by selecting bank-0 214 a through thestorage control bus 212, sending a command to clear the input buffers ofthe solid-state storage elements 110 associated with the bank-0 214 a,and sending a command to validate the status of the solid-state storageelements 216, 218, 220 associated with the bank-0 214 a. The storage buscontroller 348 then transmits a write subcommand on the storage I/O bus210, which contains the physical addresses including the address of thelogical erase block for each individual physical erase solid-stagestorage element 216 a-m as mapped from the logical erase block address.The storage bus controller 348 then muxes the write buffer 320 throughthe write sync buffer 308 to the storage I/O bus 210 through the MUX 350and streams write data to the appropriate page. When the page is full,then storage bus controller 348 causes the solid-state storage elements216 a-m associated with the bank-0 214 a to program the input buffer tothe memory cells within the solid-state storage elements 216 a-m.Finally, the storage bus controller 348 validates the status to ensurethat page was correctly programmed.

A read operation is similar to the write example above. During a readoperation, typically the bus arbiter 420, or other component of the bankinterleave controller 344, receives data and corresponding statusinformation and sends the data to the read data pipeline 108 whilesending the status information on to the control and status registers340. Typically, a read data command forwarded from bus arbiter 420 tothe storage bus controller 348 will cause the MUX 350 to gate the readdata on storage I/O bus 210 to the read data pipeline 108 and sendstatus information to the appropriate control and status registers 340through the status MUX 422.

The bus arbiter 420 coordinates the various command types and dataaccess modes so that only an appropriate command type or correspondingdata is on the bus at any given time. If the bus arbiter 420 hasselected a write command, and write subcommands and corresponding dataare being written to the solid-state storage 110, the bus arbiter 420will not allow other command types on the storage I/O bus 210.Beneficially, the bus arbiter 420 uses timing information, such aspredicted command execution times, along with status informationreceived concerning bank 214 status to coordinate execution of thevarious commands on the bus with the goal of minimizing or eliminatingidle time of the busses.

The master controller 224 through the bus arbiter 420 typically usesexpected completion times of the commands stored in the queues 410, 412,414, 416, along with status information, so that when the subcommandsassociated with a command are executing on one bank 214 a, othersubcommands of other commands are executing on other banks 214 b-n. Whenone command is fully executed on a bank 214 a, the bus arbiter 420directs another command to the bank 214 a. The bus arbiter 420 may alsocoordinate commands stored in the queues 410, 412, 414, 416 with othercommands that are not stored in the queues 410, 412, 414, 416.

For example, an erase command may be sent out to erase a group of eraseblocks within the solid-state storage 110. An erase command may take 10to 1000 times more time to execute than a write or a read command or 10to 100 times more time to execute than a program command. For N banks214, the bank interleave controller 344 may split the erase command intoN commands, each to erase a virtual erase block of a bank 214 a. Whilebank-0 214 a is executing an erase command, the bus arbiter 420 mayselect other commands for execution on the other banks 214 b-n. The busarbiter 420 may also work with other components, such as the storage buscontroller 348, the master controller 224, etc., to coordinate commandexecution among the buses. Coordinating execution of commands using thebus arbiter 420, bank controllers 418, queues 410, 412, 414, 416, andagents 402, 404, 406, 408 of the bank interleave controller 344 candramatically increase performance over other solid-state storage systemswithout a bank interleave function.

In one embodiment, the solid-state controller 104 includes one bankinterleave controller 344 that serves all of the storage elements 216,218, 220 of the solid-state storage 110. In another embodiment, thesolid-state controller 104 includes a bank interleave controller 344 foreach row of storage elements 216 a-m, 218 a-m, 220 a-m. For example, onebank interleave controller 344 serves one row of storage elements SSS0.0-SSS 0.N 216 a, 218 a, 220 a, a second bank interleave controller 344serves a second row of storage elements SSS 1.0-SSS 1.N 216 b, 218 b,220 b, etc.

FIG. 4B is a schematic block diagram illustrating an alternateembodiment 401 of a bank interleave controller 344 in the solid-statestorage controller 104 in accordance with the present invention. Thecomponents 210, 212, 340, 346, 348, 350, 402-430 depicted in theembodiment shown in FIG. 4B are substantially similar to the bankinterleave apparatus 400 described in relation to FIG. 4A except thateach bank 214 includes a single queue 432 a-n and the read commands,write commands, erase commands, management commands, etc. for a bank(e.g. Bank-0 214 a) are directed to a single queue 432 a for the bank214 a. The queues 432, in one embodiment, are FIFO. In anotherembodiment, the queues 432 can have commands pulled from the queues 432in an order other than the order they were stored. In another alternateembodiment (not shown), the read agent 402, write agent 404, erase agent406, and management agent 408 may be combined into a single agentassigning commands to the appropriate queues 432 a-n.

In another alternate embodiment (not shown), commands are stored in asingle queue where the commands may be pulled from the queue in an orderother than how they are stored so that the bank interleave controller344 can execute a command on one bank 214 a while other commands areexecuting on the remaining banks 214 b-n. One of skill in the art willeasily recognize other queue configurations and types to enableexecution of a command on one bank 214 a while other commands areexecuting on other banks 214 b-n.

Storage-Specific Components

The solid-state storage controller 104 includes a synchronization buffer346 that buffers commands and status messages sent and received from thesolid-state storage 110. The synchronization buffer 346 is located atthe boundary between the solid-state storage clock domain and the localbus clock domain and provides buffering to account for the clock domaindifferences. The synchronization buffer 346, write synchronizationbuffer 308, and read synchronization buffer 328 may be independent ormay act together to buffer data, commands, status messages, etc. In thepreferred embodiment, the synchronization buffer 346 is located wherethere are the fewest number of signals crossing the clock domains. Oneskilled in the art will recognize that synchronization between clockdomains may be arbitrarily moved to other locations within thesolid-state storage device 102 in order to optimize some aspect ofdesign implementation.

The solid-state storage controller 104 includes a storage bus controller348 that interprets and translates commands for data sent to and readfrom the solid-state storage 110 and status messages received from thesolid-state storage 110 based on the type of solid-state storage 110.For example, the storage bus controller 348 may have different timingrequirements for different types of storage, storage with differentperformance characteristics, storage from different manufacturers, etc.The storage bus controller 348 also sends control commands to thestorage control bus 212.

In the preferred embodiment, the solid-state storage controller 104includes a MUX 350 that comprises an array of multiplexers 350 a-n whereeach multiplexer is dedicated to a row in the solid-state storage array110. For example, multiplexer 350 a is associated with solid-statestorage elements 216 a, 218 a, 220 a. MUX 350 routes the data from thewrite data pipeline 106 and commands from the storage bus controller 348to the solid-state storage 110 via the storage I/O bus 210 and routesdata and status messages from the solid-state storage 110 via thestorage I/O bus 210 to the read data pipeline 108 and the control andstatus registers 340 through the storage bus controller 348,synchronization buffer 346, and bank interleave controller 344.

In the preferred embodiment, the solid-state storage controller 104includes a MUX 350 for each row of solid-state storage elements (e.g.SSS 0.1 216 a, SSS 0.2 218 a, SSS 0.N 220 a). A MUX 350 combines datafrom the write data pipeline 106 and commands sent to the solid-statestorage 110 via the storage I/O bus 210 and separates data to beprocessed by the read data pipeline 108 from commands. Packets stored inthe write buffer 320 are directed on busses out of the write buffer 320through a write synchronization buffer 308 for each row of solid-statestorage elements (SSS x.0 to SSS x.N 216, 218, 220) to the MUX 350 foreach row of solid-state storage elements (SSS x.0 to SSS x.N 216, 218,220). The commands and read data are received by the MUXes 350 from thestorage I/O bus 210. The MUXes 350 also direct status messages to thestorage bus controller 348.

The storage bus controller 348 includes a mapping module 424. Themapping module 424 maps a logical address of an erase block to one ormore physical addresses of an erase block. For example, a solid-statestorage 110 with an array of twenty storage elements (e.g. SSS 0.0 toSSS M.0 216) per block 214 a may have a logical address for a particularerase block mapped to twenty physical addresses of the erase block, onephysical address per storage element. Because the storage elements areaccessed in parallel, erase blocks at the same position in each storageelement in a row of storage elements 216 a, 218 a, 220 a will share aphysical address. To select one erase block (e.g. in storage element SSS0.0 216 a) instead of all erase blocks in the row (e.g. in storageelements SSS 0.0, 0.1, . . . 0.N 216 a, 218 a, 220 a), one bank (in thiscase bank-0 214 a) is selected.

This logical-to-physical mapping for erase blocks is beneficial becauseif one erase block becomes damaged or inaccessible, the mapping can bechanged to map to another erase block. This mitigates the loss of losingan entire virtual erase block when one element's erase block is faulty.The remapping module 430 changes a mapping of a logical address of anerase block to one or more physical addresses of a virtual erase block(spread over the array of storage elements). For example, virtual eraseblock 1 may be mapped to erase block 1 of storage element SSS 0.0 216 a,to erase block 1 of storage element SSS 1.0 216 b, . . . , and tostorage element M.0 216 m, virtual erase block 2 may be mapped to eraseblock 2 of storage element SSS 0.1 218 a, to erase block 2 of storageelement SSS 1.1 218 b, . . . , and to storage element M.1 218 m, etc.

If erase block 1 of a storage element SSS0.0 216 a is damaged,experiencing errors due to wear, etc., or cannot be used for somereason, the remapping module 430 could change the logical-to-physicalmapping for the logical address that pointed to erase block 1 of virtualerase block 1. If a spare erase block (call it erase block 221) ofstorage element SSS 0.0 216 a is available and currently not mapped, theremapping module 430 could change the mapping of virtual erase block 1to point to erase block 221 of storage element SSS 0.0 216 a, whilecontinuing to point to erase block 1 of storage element SSS 1.0 216 b,erase block 1 of storage element SSS 2.0 (not shown) . . . , and tostorage element M.0 216 m. The mapping module 424 or remapping module430 could map erase blocks in a prescribed order (virtual erase block 1to erase block 1 of the storage elements, virtual erase block 2 to eraseblock 2 of the storage elements, etc.) or may map erase blocks of thestorage elements 216, 218, 220 in another order based on some othercriteria.

In one embodiment, the erase blocks could be grouped by access time.Grouping by access time, meaning time to execute a command, such asprogramming (writing) data into pages of specific erase blocks, canlevel command completion so that a command executed across the eraseblocks of a virtual erase block is not limited by the slowest eraseblock. In other embodiments, the erase blocks may be grouped by wearlevel, health, etc. One of skill in the art will recognize other factorsto consider when mapping or remapping erase blocks.

In one embodiment, the storage bus controller 348 includes a statuscapture module 426 that receives status messages from the solid-statestorage 110 and sends the status messages to the status MUX 422. Inanother embodiment, when the solid-state storage 110 is flash memory,the storage bus controller 348 includes a NAND bus controller 428. TheNAND bus controller 428 directs commands from the read and write datapipelines 106, 108 to the correct location in the solid-state storage110, coordinates timing of command execution based on characteristics ofthe flash memory, etc. If the solid-state storage 110 is anothersolid-state storage type, the NAND bus controller 428 would be replacedby a bus controller specific to the storage type. One of skill in theart will recognize other functions of a NAND bus controller 428.

Flow Charts

FIG. 5A is a schematic flow chart diagram illustrating one embodiment ofa method 500 for managing data in a solid-state storage device 102 usinga data pipeline in accordance with the present invention. The method 500begins 502 and the input buffer 306 receives 504 one or more datasegments to be written to the solid-state storage 110. The one or moredata segments typically include at least a portion of an object but maybe an entire object. The packetizer 302 may create one or more objectspecific packets in conjunction with an object. The packetizer 302 addsa header to each packet which typically includes the length of thepacket and a sequence number for the packet within the object. Thepacketizer 302 receives 504 the one or more data or metadata segmentsthat were stored in the input buffer 306 and packetizes 506 the one ormore data or metadata segments by creating one or more packets sized forthe solid-state storage 110 where each packet includes one header anddata from the one or more segments.

Typically, a first packet includes an object identifier that identifiesthe object for which the packet was created. A second packet may includea header with information used by the solid-state storage device 102 toassociate the second packet to the object identified in the first packetand offset information locating the second packet within the object, anddata. The solid-state storage device controller 202 manages the bank 214and physical area to which the packets are streamed.

The ECC generator 304 receives a packet from the packetizer 302 andgenerates 508 ECC for the data packets. Typically, there is no fixedrelationship between packets and ECC blocks. An ECC block may compriseone or more packets. A packet may comprise one or more ECC blocks. Apacket may start and end anywhere within an ECC block. A packet maystart anywhere in a first ECC block and end anywhere in a subsequent ECCblock.

The write synchronization buffer 308 buffers 510 the packets asdistributed within the corresponding ECC blocks prior to writing ECCblocks to the solid-state storage 110 and then the solid-state storagecontroller 104 writes 512 the data at an appropriate time consideringclock domain differences, and the method 500 ends 514. The write synchbuffer 308 is located at the boundary between a local clock domain and asolid-state storage 110 clock domain. Note that the method 500 describesreceiving one or more data segments and writing one or more data packetsfor convenience, but typically a stream of data segments is received asa group. Typically a number of ECC blocks comprising a complete virtualpage of solid-state storage 110 are written to the solid-state storage110. Typically the packetizer 302 receives data segments of one size andgenerates packets of another size. This necessarily requires data ormetadata segments or parts of data or metadata segments to be combinedto form data packets to capture all of the data of the segments intopackets.

FIG. 5B is a schematic flow chart diagram illustrating one embodiment ofa method for in-server SAN in accordance with the present invention. Themethod 500 begins 552 and the storage communication module 162facilitates 554 communication between a first storage controller 152 aand at least one device external to the first server 112 a. Thecommunication between the first storage controller 152 a and theexternal device is independent from the first server 112 a. The firststorage controller 112 a is within the first server 112 a and the firststorage controller 152 a controls at least one storage device 154 a. Thefirst server 112 a includes a network interface 156 a collocated withthe first server 112 a and the first storage controller 152 a. Thein-server SAN module 164 services 556 a storage request and the method501 ends 558. The in-server SAN module services 556 the storage requestusing a network protocol and/or a bus protocol. The in-server SAN module164 services 556 the storage request independent from the first server112 a and the service request is received from a client 114, 114 a.

FIG. 6 is a schematic flow chart diagram illustrating another embodimentof a method 600 for managing data in a solid-state storage device 102using a data pipeline in accordance with the present invention. Themethod 600 begins 602 and the input buffer 306 receives 604 one or moredata or metadata segments to be written to the solid-state storage 110.The packetizer 302 adds a header to each packet which typically includesthe length of the packet within the object. The packetizer 302 receives604 the one or more segments that were stored in the input buffer 306and packetizes 606 the one or more segments by creating one or morepackets sized for the solid-state storage 110 where each packet includesa header and data from the one or more segments.

The ECC generator 304 receives a packet from the packetizer 302 andgenerates 608 one or more ECC blocks for the packets. The writesynchronization buffer 308 buffers 610 the packets as distributed withinthe corresponding ECC blocks prior to writing ECC blocks to thesolid-state storage 110 and then the solid-state storage controller 104writes 612 the data at an appropriate time considering clock domaindifferences. When data is requested from the solid-state storage 110,ECC blocks comprising one or more data packets are read into the readsynchronization buffer 328 and buffered 614. The ECC blocks of thepacket are received over the storage I/O bus 210. Since the storage I/Obus 210 is bi-directional, when data is read, write operations, commandoperations, etc. are halted.

The ECC correction module 322 receives the ECC blocks of the requestedpackets held in the read synchronization buffer 328 and corrects 616errors within each ECC block as necessary. If the ECC correction module322 determines that one or more errors exist in an ECC block and theerrors are correctable using the ECC syndrome, the ECC correction module322 corrects 616 the error in the ECC block. If the ECC correctionmodule 322 determines that a detected error is not correctable using theECC, the ECC correction module 322 sends an interrupt.

The depacketizer 324 receives 618 the requested packet after the ECCcorrection module 322 corrects any errors and depacketizes 618 thepackets by checking and removing the packet header of each packet. Thealignment module 326 receives packets after depacketizing, removesunwanted data, and re-formats 620 the data packets as data or metadatasegments of an object in a form compatible with the device requestingthe segment or object. The output buffer 330 receives requested packetsafter depacketizing and buffers 622 the packets prior to transmission tothe requesting device 155, and the method 600 ends 624.

FIG. 7 is a schematic flow chart diagram illustrating an embodiment of amethod 700 for managing data in a solid-state storage device 102 using abank interleave in accordance with the present invention. The method 700begins 702 and the bank interleave controller 344 directs 704 one ormore commands to two or more queues 410, 412, 414, 416. Typically theagents 402, 404, 406, 408 direct 704 the commands to the queues 410,412, 414, 416 by command type. Each set of queues 410, 412, 414, 416includes a queue for each command type. The bank interleave controller344 coordinates 706 among the banks 214 execution of the commands storedin the queues 410, 412, 414, 416 so that a command of a first typeexecutes on one bank 214 a while a command of a second type executes ona second bank 214 b, and the method 700 ends 708.

Storage Space Recovery

FIG. 8 is a schematic block diagram illustrating one embodiment of anapparatus 800 for garbage collection in a solid-state storage device 102in accordance with the present invention. The apparatus 800 includes asequential storage module 802, a storage division selection module 804,a data recovery module 806, and a storage division recovery module 808,which are described below. In other embodiments, the apparatus 800includes a garbage marking module 812 and an erase module 810.

The apparatus 800 includes a sequential storage module 802 thatsequentially writes data packets in a page within a storage division.The packets are sequentially stored whether they are new packets ormodified packets. Modified packets are in this embodiment are typicallynot written back to a location where they were previously stored. In oneembodiment, the sequential storage module 802 writes a packet to a firstlocation in a page of a storage division, then to the next location inthe page, and to the next, and the next, until the page is filled. Thesequential storage module 802 then starts to fill the next page in thestorage division. This continues until the storage division is filled.

In a preferred embodiment, the sequential storage module 802 startswriting packets to storage write buffers in the storage elements (e.g.SSS 0.0 to SSS M.0 216) of a bank (bank-0 214 a). When the storage writebuffers are full, the solid-state storage controller 104 causes the datain the storage write buffers to be programmed into designated pageswithin the storage elements 216 of the bank 214 a. Then another bank(e.g. bank-1 214b) is selected and the sequential storage module 802starts writing packets to storage write buffers of the storage elements218 of the bank 214 b while the first bank-0 is programming thedesignated pages. When the storage write buffers of this bank 214 b arefull, the contents of the storage write buffers are programmed intoanother designated page in each storage element 218. This process isefficient because while one bank 214 a is programming a page, storagewrite buffers of another bank 214 b can be filling.

The storage division includes a portion of a solid-state storage 110 ina solid-state storage device 102. Typically the storage division is anerase block. For flash memory, an erase operation on an erase blockwrites ones to every bit in the erase block by charging each cell. Thisis a lengthy process compared to a program operation which starts with alocation being all ones, and as data is written, some bits are changedto zero by discharging the cells written with a zero. However, where thesolid-state storage 110 is not flash memory or has flash memory where anerase cycle takes a similar amount of time as other operations, such asa read or a program, the storage division may not be required to beerased.

As used herein, a storage division is equivalent in area to an eraseblock but may or may not be erased. Where erase block is used herein, anerase block may refer to a particular area of a designated size within astorage element (e.g. SSS 0.0 216 a) and typically includes a certainquantity of pages. Where “erase block” is used in conjunction with flashmemory, it is typically a storage division that is erased prior to beingwritten. Where “erase block” is used with “solid-state storage,” it mayor may not be erased. As used herein, an erase block may include oneerase block or a group of erase blocks with one erase block in each of arow of storage elements (e.g. SSS 0.0 to SSS M.0 216 a-n), which mayalso be referred to herein as a virtual erase block. When referring tothe logical construct associated with the virtual erase block, the eraseblocks may be referred to herein as a logical erase block (“LEB”).

Typically, the packets are sequentially stored by order of processing.In one embodiment, where a write data pipeline 106 is used, thesequential storage module 802 stores packets in the order that they comeout of the write data pipeline 106. This order may be a result of datasegments arriving from a requesting device 155 mixed with packets ofvalid data that are being read from another storage division as validdata is being recovered from a storage division during a recoveryoperation as explained below. Re-routing recovered, valid data packetsto the write data pipeline 106 may include the garbage collector bypass316 as described above in relation to the solid-state storage controller104 of FIG. 3.

The apparatus 800 includes a storage division selection module 804 thatselects a storage division for recovery. Selecting a storage divisionfor recovery may be to reuse the storage division by the sequentialstorage module 802 for writing data, thus adding the recovered storagedivision to the storage pool, or to recover valid data from the storagedivision after determining that the storage division is failing,unreliable, should be refreshed, or other reason to take the storagedivision temporarily or permanently out of the storage pool. In anotherembodiment, the storage division selection module 804 selects a storagedivision for recovery by identifying a storage division or erase blockwith a high amount of invalid data.

In another embodiment, the storage division selection module 804 selectsa storage division for recovery by identifying a storage division orerase block with a low amount of wear. For example, identifying astorage division or erase block with a low amount of wear may includeidentifying a storage division with a low amount of invalid data, a lownumber of erase cycles, low bit error rate, or low program count (lownumber of times a page of data in a buffer is written to a page in thestorage division; program count may be measured from when the device wasmanufactured, from when the storage division was last erased, from otherarbitrary events, and from combinations of these). The storage divisionselection module 804 may also use any combination of the above or otherparameters to determine a storage division with a low amount of wear.Selecting a storage division for recovery by determining a storagedivision with a low amount of wear may be desirable to find storagedivisions that are under used, may be recovered for wear leveling, etc.

In another embodiment, the storage division selection module 804 selectsa storage division for recovery by identifying a storage division orerase block with a high amount of wear. For example, identifying astorage division or erase block with a high amount of wear may includeidentifying a storage division with a high number of erase cycles, highbit error rate, a storage division with a non-recoverable ECC block, orhigh program count. The storage division selection module 804 may alsouse any combination of the above or other parameters to determine astorage division with a high amount of wear. Selecting a storagedivision for recovery by determining a storage division with a highamount of wear may be desirable to find storage divisions that are overused, may be recovered by refreshing the storage division using an erasecycle, etc. or to retire the storage division from service as beingunusable.

The apparatus 800 includes a data recovery module 806 that reads validdata packets from the storage division selected for recovery, queues thevalid data packets with other data packets to be written sequentially bythe sequential storage module 802, and updates an index with a newphysical address of the valid data written by the sequential storagemodule 802. Typically, the index is the object index mapping data objectidentifiers of objects to physical addresses of where packets derivedfrom the data object are stored in the solid-state storage 110.

In one embodiment the apparatus 800 includes a storage division recoverymodule 808 that prepares the storage division for use or reuse and marksthe storage division as available to the sequential storage module 802for sequentially writing data packets after the data recovery module 806has completed copying valid data from the storage division. In anotherembodiment, the apparatus 800 includes a storage division recoverymodule 808 that marks the storage division selected for recovery asunavailable for storing data. Typically this is due to the storagedivision selection module 804 identifying a storage division or eraseblock with a high amount of wear such that the storage division or eraseblock is not in condition to be used for reliable data storage.

In one embodiment, the apparatus 800 is in a solid-state storage devicecontroller 202 of a solid-state storage device 102. In anotherembodiment, the apparatus 800 controls a solid-state storage devicecontroller 202. In another embodiment, a portion of the apparatus 800 isin a solid-state storage device controller 202. In another embodiment,the object index updated by the data recovery module 806 is also locatedin the solid-state storage device controller 202

In one embodiment, the storage division is an erase block and theapparatus 800 includes an erase module 810 that erases an erase blockselected for recovery after the data recovery module 806 has copiedvalid data packets from the selected erase block and before the storagedivision recovery module 808 marks the erase block as available. Forflash memory and other solid-state storage with an erase operationtaking much longer than read or write operations, erasing a data blockprior to making it available for writing new data is desirable forefficient operation. Where the solid-state storage 110 is arranged inbanks 214, the erase operation by the erase module 810 may be executedon one bank while other banks are executing reads, writes, or otheroperations.

In one embodiment, the apparatus 800 includes a garbage marking module812 that identifies a data packet in a storage division as invalid inresponse to an operation indicating that the data packet is no longervalid. For example, if a data packet is deleted, the garbage markingmodule 812 may identify the data packet as invalid. A read-modify-writeoperation is another way for a data packet to be identified as invalid.In one embodiment, the garbage marking module 812 may identify the datapacket as invalid by updating an index.

In another embodiment, the garbage marking module 812 may identify thedata packet as invalid by storing another data packet that indicatesthat the invalid data packet has been deleted. This is advantageousbecause storing, in the solid-state storage 110, information that thedata packet has been deleted allows the object index reconstructionmodule 262 or similar module to reconstruct the object index with anentry indicating that the invalid data packet has been deleted.

In one embodiment, the apparatus 800 may be utilized to fill theremainder of a virtual page of data following a flush command in orderto improve overall performance, where the flush command halts dataflowing into the write pipeline 106 until the write pipeline 106 emptiesand all packets have been permanently written into non-volatilesolid-state storage 110. This has the benefit of reducing the amount ofgarbage collection required, the amount of time used to erase storagedivisions, and the amount of time required to program virtual pages. Forexample, a flush command may be received when only one small packet isprepared for writing into the virtual page of the solid-state storage100. Programming this nearly empty virtual page might result in a needto immediately recover the wasted space, causing the valid data withinthe storage division to be unnecessarily garbage collected and thestorage division erased, recovered and returned to the pool of availablespace for writing by the sequential storage module 802.

Marking the data packet as invalid rather than actually erasing aninvalid data packet is efficient because, as mentioned above, for flashmemory and other similar storage an erase operation takes a significantamount of time. Allowing a garbage collection system, as described inthe apparatus 800, to operate autonomously within the solid-statestorage 110 provides a way to separate erase operations from reads,writes, and other faster operations so that the solid-state storagedevice 102 can operate much faster than many other solid-state storagesystems or data storage devices.

FIG. 9 is a schematic flow chart diagram illustrating an embodiment of amethod 900 for storage recovery in accordance with the presentinvention. The method 900 begins 902 and the sequential storage module802 sequentially writes 904 data packets in a storage division. Thestorage division is a portion of a solid-state storage 110 in asolid-state storage device 102. Typically a storage division is an eraseblock. The data packets are derived from an object and the data packetsare sequentially stored by order of processing.

The storage division selection module 804 selects 906 a storage divisionfor recovery and the data recovery module 806 reads 908 valid datapackets from the storage division selected for recovery. Typically validdata packets are data packets that have not been marked for erasure ordeletion or some other invalid data marking and are considered valid or“good” data. The data recovery module 806 queues 910 the valid datapackets with other data packets scheduled to be written sequentially bythe sequential storage module 802. The data recovery module 806 updates912 an index with a new physical address of the valid data written bythe sequential storage module 802. The index includes a mapping ofphysical addresses of data packets to object identifiers. The datapackets are those stored in stored in the solid-state storage 110 andthe object identifiers correspond to the data packets.

After the data recovery module 806 completes copying valid data from thestorage division, the storage division recovery module 808 marks 914 thestorage division selected for recovery as available to the sequentialstorage module 802 for sequentially writing data packets and the method900 ends 916.

Progressive Raid

FIG. 10 is a schematic block diagram illustrating one embodiment of asystem 1600 for progressive RAID in accordance with the presentinventions. The system 1600 includes N storage devices 150 and Mparity-mirror storage devices 1602 accessible through a computer network116 by one or more clients 114. The N storage devices 150 andparity-mirror storage devices 1602 may be located in one or more servers112. The storage devices 150, servers 112, computer network 116, andclients 114 are substantially similar to those described above. Theparity-mirror devices 1602 are typically similar or identical to the Nstorage devices 150 and are typically designated as a parity-mirrorstorage device 1602 for a stripe.

In one embodiment, the N storage devices 150 and M parity-mirror storagedevices 1602 are included in or accessible through one server 112 andmay be networked together using a system bus, SAN, or other computernetwork 116. In another embodiment, the N storage devices 150 and Mparity-mirror storage devices 1602 are located in or accessible throughmultiple servers 112 a-n+m. For example, the storage devices 150 andparity-mirror storage devices 1602 may be part of an in-server SAN asdescribed above in relation to the system 103 of FIG. 1C and the method105 of FIG. 5B.

In one embodiment, a parity-mirror storage device 1602 stores all paritydata segments of the stripes stored in the progressive RAID. In anotherpreferred embodiment, a storage device 150 of the storage device set1604 assigned to the progressive RAID is assigned to be a parity-mirrorstorage device 1602 for a particular stripe and the assignment isrotated so that the parity data segments are rotated, for each stripe,among the N+M storage devices 150. This embodiment, offers a performanceadvantage over assigning a single storage device 150 to be aparity-mirror storage device 1602 for each stripe. By rotating theparity-mirror storage device 1602, the overhead associated withcalculating and storing parity data segments can be distributed.

In one embodiment, the storage devices 150 are solid-state storagedevices 102, each with associated solid-state storage 110 and asolid-state storage controller 104. In another embodiment, each storagedevice 150 includes a solid-state storage controller 104 and associatedsolid-state storage 110 acts as cache for other less expensive, lowerperformance storage, such as tape storage or hard disk drives. Inanother embodiment, one or more of the servers 112 include one or moreclients 114 that send storage requests to the progressive RAID. One ofskill in the art will recognize other system configurations with Nstorage devices 150 and one or more parity-mirror storage devices 1602that may be configured for progressive RAID.

FIG. 11 is a schematic block diagram illustrating one embodiment of anapparatus 1700 for progressive RAID in accordance with the presentinvention. The apparatus 1700 includes, in various embodiments, astorage request receiver module 1702, a striping module 1704, aparity-mirror module 1706, and a parity progression module 1708, aparity alteration module 1710, a mirrored set module 1712, an updatemodule 1714, a mirror restoration module 1716 with a direct clientresponse module 1718, a pre-consolidation module 1720, apost-consolidation module 1722, a data rebuild module 1724, and a parityrebuild module 1726, which are described below. The modules 1702-1726are depicted in a server 112, but some or all of the functions of themodules 1702-1726 may also be distributed in multiple servers 112,storage controllers 152, storage devices 150, clients 114, etc.

The apparatus 1700 includes a storage request receiver module 1702 thatreceives a request to store data, where the data is data of a file or ofan object. In one embodiment, the storage request is an object request.In another embodiment, the storage request is a block storage request.The storage request in one embodiment, does not include data, butincludes commands that can be used by the storage devices 150 andparity-mirror storage devices 1602 to DMA or RDMA data from a client 114or other source. In another embodiment, the storage request includesdata to be stored as a result of the storage request. In anotherembodiment, the storage request includes one command capable of havingthe data stored in the storage device set 1604. In another embodiment,the storage request includes multiple commands. One of skill in the artwill recognize other storage requests to store data appropriate forprogressive RAID.

The data is stored in a location accessible to the apparatus 1700. Inone embodiment, the data is available in a random access memory (“RAM”),such as a RAM used by the client 114 or server. In another embodiment,the data is stored in a hard disk drive, tape storage, or other massstorage device. In one embodiment, the data is configured as an objector as a file. In another embodiment, the data is configured as a datablock which is part of an object or a file. One of skill in the art willrecognize other forms and locations for the data that is the subject ofthe storage request.

The apparatus 1700 includes a striping module 1704 that calculates astripe pattern for the data. The stripe pattern includes one or morestripes, where each stripe includes a set of N data segments. Typicallythe number of data segments in a stripe depends on how many storagedevices 150 are assigned to the RAID group. For example, if RAID 5 isused, one storage device 150 is assigned as a parity-mirror storagedevice 1602 a to store parity data for a particular stripe. If fourother storage devices 150 a, 150 b, 150 c, 150 d are assigned to theRAID group, a stripe will have four data segments in addition to theparity data segment. The striping module 1704 writes N data segments toN of a stripe to N storage devices 150 a-n so that each of the N datasegments is written to a separate storage device 150 a, 150 b, . . . 150n within a set 1604 of storage devices 150 assigned to the stripe. Oneof skill in the art will appreciate various combinations of storagedevices 150 that may be assigned to a RAID group for a particular RAIDlevel and how to create a striping pattern and divide data into N datasegments per stripe.

The apparatus 1700 includes a parity-mirror module 1706 that writes aset of N data segments of the stripe to one or more parity-mirrorstorage devices 1602 within the storage device set 1604, where theparity-mirror storage devices 1602 are in addition to the N storagedevices 150. The N data segments are then available for futurecalculation of a parity data segment. Rather than immediatelycalculating the parity data segment, the parity-mirror module 1706copies the set of N data segments to the parity-mirror storage devices1602, which typically requires less time than storing the N datasegments. Once the N data segments are stored on the parity-mirrorstorage device 1602, the N data segments are available to be read orused to restore data if one of the N storage devices 150 becomesunavailable. Reading data also has the advantages of a RAID 0configuration because all of the N data segments are available togetherfrom one storage device (e.g. 1602 a). For more than one parity-mirrorstorage device (e.g. 1602 a, 1602 b), the parity-mirror module 1706copies the N data segments to each parity-mirror storage device 1602 a,1602 b.

The apparatus 1700 includes a parity progression module 1708 thatcalculates one or more parity data segments for the stripe in responseto a storage consolidation operation. The one or more parity datasegments calculated from the N data segments are stored on theparity-mirror storage devices 1602. The parity progression module 1708stores a parity data segment on each of the one or more parity-mirrorstorage devices 1602. The storage consolidation operation is conductedto recover at least storage space or data or both on at least one of theone or more parity-mirror storage devices 1602. For example, a storageconsolidation operation may be a data garbage collection on asolid-state storage device 102 as described above in relation to theapparatus 800 and method 900 of FIGS. 8 and 9. The storage consolidationoperation may also include a defragmentation operation for a hard diskdrive, or other similar operation that consolidates data to increasestorage space. The storage consolidation operation, as used herein, mayalso include an operation to recover data, for example, if a storagedevice 150 is unavailable, to recover from an error, or other reason forreading data from the parity-mirror storage device 1602. In anotherembodiment, the parity generation module 1708 simply calculates theparity data segment when the parity-mirror storage device 1602 is lessbusy.

Advantageously, by delaying calculation and storage of the parity datasegment of a stripe, the N data segments on the parity-mirror storagedevice 1602 are available for reading the data segments, recoveringdata, rebuilding data on a storage device 150 until more storage spaceis needed on the parity-mirror storage device 1602 or other reason for astorage consolidation operation. The parity progression module 1708 maythen run as a background operation, autonomously from the storagerequest receiver module 1702, the striping module 1704, or theparity-mirror module 1706. One of skill in the art will easily recognizeother reasons to delay calculation of a parity data segment as part of aprogressive RAID operation.

In one embodiment, some or all of the functions of the modules1702-1708, receiving an request to store data, calculating a stripepattern and writing N data segments to the N storage devices, writing aset of N data segments to a parity-mirror storage device, andcalculating the parity data segment, occur on a storage device 150 ofthe storage device set 1604, a client 114, and a third-party RAIDmanagement device. The third-party RAID management device may be aserver 114 or other computer.

In one embodiment, the apparatus 1700 includes a parity alternationmodule 1710 that alternates, for each stripe, which of the storagedevices 150 within the storage device set 1604 are assigned to be theone or more parity-mirror storage devices 1602 for the stripe. Asdiscussed above in relation to the system 1600 of FIG. 10, by rotatingwhich storage device 150 is used for the parity-mirror storage devicefor a stripe, the work calculation of the various parity data segmentsis spread among the storage devices 150 of a storage device set 1604.

In another embodiment, the storage device set 1604 is a first storagedevice set and the apparatus 1700 includes a mirrored set module 1712that creates one or more storage device sets in addition to the firststorage set 1604 so that each of the one or more additional storagedevice sets include at least an associated striping module 1704 thatwrites the N data segments to N storage devices 150 of each of the oneor more additional storage sets. In a related embodiment, each of theone or more additional storage device sets includes an associated aparity-mirror module 1706 for storing a set of the N data segments and aparity progression module 1708 for calculating one or more parity datasegments. Where the mirrored set module 1712 creates one or moremirrored storage device sets, the RAID may be a nested RAID such as RAID50. In this embodiment, the RAID level may be progressed from a RAID 10where data is striped and mirrored, to a RAID 50 or RAID 60, where aparity data segment is calculated and stored for each storage device set1604.

In one embodiment, the apparatus 1700 includes and update module 1714.The update module 1714 is typically used where the N data segments on aparity-mirror storage device 1602 have not been progressed to a paritydata segment. The update module 1714 receives an updated data segment,where the updated data segment corresponds to an existing data segmentof the N data segments stored on the N storage devices 150. The updatemodule 1714 copies the updated data segment to the storage device 150 ofthe stripe where the existing data segment is stored and to the one ormore parity-mirror storage devices 1602 of the stripe. The update module1714 replaces the existing data segment stored on the storage device 150of the N storage devices 150 a-n with the updated data segment andreplaces the corresponding existing data segment stored on the one ormore parity-mirror storage devices 1602 with the updated data segment.

In one embodiment replacing a data segment includes writing the datasegment to a storage device 150 and then marking a corresponding datasegment as invalid for subsequent garbage collection. An example of thisembodiment is described for solid-state storage 110 and the garbagecollection apparatus described above in relation to FIGS. 8 and 9. Inanother embodiment, replacing a data segment includes overwriting anexisting data segment with an updated data segment.

In one embodiment, the set of storage devices 1604 is a first storagedevice set and the apparatus 1700 includes a mirror restoration module1716 that recovers a data segment stored on a storage device 150 of thefirst storage set 1604 where the storage device 150 of the first storageset 1604 is unavailable. The data segment is recovered from a mirrorstorage device containing a copy of the data segment. The mirror storagedevice includes one of a set of one or more storage devices 150 thatstores a copy of the N data segments.

In a further embodiment, the mirror restoration module 1716 recovers thedata segment in response to a read request from a client 114 to read thedata segment. In another related embodiment, the mirror restorationmodule 1716 also includes a direct client response module 1718 thatsends the requested data segment to the client 114 from the mirrorstorage device. In this embodiment, the requested data segment is copiedto the client 114 so that the client 114 does not have to wait until thedata segment is recovered before transmitting the data segment on to theclient 114.

In one embodiment, the apparatus 1700 includes a pre-consolidationrestoration module 1720 that recovers a data segment stored on a storagedevice 150 of the storage set 1604 in response to a request to read thedata segment. In the embodiment the storage device 150 is unavailableand the data segment is recovered from the a parity-mirror storagedevice 1602 prior to the parity progression module 1708 generating theone or more parity data segments on the one or more parity-mirrorstorage devices 1602.

In another embodiment, the apparatus 1700 includes a post-consolidationrestoration module 1724 that recovers a data segment stored on a storagedevice 150 of the storage set. In the embodiment, the storage device 150is unavailable and the data segment is recovered using one or moreparity data segments stored on one or more of the parity-mirror storagedevices 150 after the parity progression module 1708 generates the oneor more parity data segments. For example, the post-consolidationrestoration module 1724 uses a parity data segment and available datasegments on the available N storage devices 150 to recreate the missingdata segment.

In one embodiment, the apparatus 1700 includes a data rebuild module1724 that stores a recovered data segment on a replacement storagedevice in a rebuild operation, where the recovered data segment matchesan unavailable data segment stored on an unavailable storage device 150.The unavailable storage device 150 is one of the N storage devices 150of the storage device set 1602. Typically, the rebuild operation occursafter a failure of the storage device 150 that stores the unavailabledata segment. The rebuild operation is to restore data segments onto thereplacement storage device to match data segments stored previously onthe unavailable storage device 150.

The data segment may be recovered for the rebuild operation from severalsources. For example, the data segment may be recovered from aparity-mirror storage device 1602 prior to progression if the matchingdata segment resides on the parity-mirror storage device 1602. Inanother example, the data segment may be recovered from a mirror storagedevice containing a copy of the unavailable data segment. Typically thedata segment is recovered from the mirror storage device if therecovered data segment does not reside on the one or more parity-mirrorstorage devices 1602, but may be recovered from the mirror storagedevice even if the matching data segment is available on the mirrorstorage device.

In another example, a regenerated data segment is regenerated from oneor more parity data segments and available data segments of the N datasegments if the recovered data segment does not reside on aparity-mirror storage device 1604 or the mirror storage device.Typically the missing data segment is regenerated only if it does notexist on another storage device 150 in some form.

In another embodiment, the apparatus 1700 includes a parity rebuildmodule 1726 that rebuilds a recovered parity data segment on areplacement storage device in a parity rebuild operation where therecovered parity data segment matches an unavailable parity data segmentstored on an unavailable parity-mirror storage device. The unavailableparity-mirror storage device is one of the one or more parity-mirrorstorage devices 1602. The parity rebuild operation restores parity datasegments onto the replacement storage device to match parity datasegments stored previously on the unavailable parity-mirror storagedevice.

To regenerate the recovered parity data segment in the rebuildoperation, data used for the rebuild may be from various sources. In oneexample, the recovered parity data segment is recovered using a paritydata segment stored on a parity-mirror storage device 1602 in a secondset of storage devices 150 storing a mirror copy of the stripe. Where amirror copy is available, using the mirrored parity data segment isdesirable because the recovered parity data segment does not have to berecalculated. In another example, the recovered parity data segment isregenerated from the N data segments stored on one of the N storagedevices 150 if the N data segments are available on the N storagedevices. Typically, the N data segments would be available on the Nstorage devices 150 where a single failure occurs on the parity-mirrorstorage device 1602 being rebuilt.

In another example, the recovered parity data segment is regeneratedfrom one or more storage devices 150 of the second set of storagedevices 150 storing copies of the N data segments if one or more of theN data segments are unavailable from the N storage devices 150 of thefirst storage device set 1604 and a matching parity data segment is notavailable on the second set of storage devices 150. In yet anotherexample, the recovered parity data segment is regenerated from theavailable data segments and non-matching parity data segments regardlessof their location within the one or more sets of storage devices 150.

Where the parity-mirror storage device is alternated among the storagedevices 150 of the storage device set 1604, typically the data rebuildmodule 1724 and the parity rebuild module 1726 act in conjunction torebuild data segments and parity data segments on a rebuilt storagedevice 150. Where a second parity-mirror storage device 1602 b isavailable, the data rebuild module 1724 and the parity rebuild module1726 can rebuild two storage devices after a failure of two storagedevices 150, 1602 of the storage device set 1604. Where a parity-mirrorstorage device 1602 is not progressed to create a parity-mirror datasegment, recovery of a data segment or a storage device 150 is quickerthan if the parity-mirror storage device 1602 is progressed and theparity data segment for a stripe has been calculated and stored and theN data segments on the parity-mirror storage device 1602 used tocalculate the parity data segment have been deleted.

FIG. 12 is a schematic block diagram illustrating one embodiment of anapparatus 1800 for updating a data segment using progressive RAID inaccordance with the present invention. Typically, the apparatus 1800pertains to a RAID group where one or more of the parity-mirror storagedevices 1602 have been progressed and include a parity data segment andnot the N data segments used to create the parity data segment. Theapparatus 1800 includes an update receiver module 1802, an update copymodule 1804, a parity update module 1806, which are described below. Themodules 1802-1806 of the apparatus 1800 are depicted in a server 112,but may be in a storage device 150, a client 114, or any combination ofdevices, or may be distributed among several devices.

A stripe, data segments, storage devices 150, a storage device set 1604,parity data segments, and the one or more parity-mirror storage device1602 are substantially similar to a stripe as describe above in relationto the apparatus 1700 of FIG. 11. The apparatus 1800 includes an updatereceiver module 1802 that receives an updated data segment where theupdated data segment corresponds to an existing data segment of anexisting stripe. In another embodiment, the update receiver module 1802may also receive multiple updates and may handle the updates together orseparately.

The apparatus 1800 includes an update copy module 1804 that copies theupdated data segment to the storage device 150 where the correspondingexisting data segment is stored and to the one or more parity-mirrorstorage devices 1602 corresponding to the existing stripe. In anotherembodiment, the update copy module 1804 copies the updated data segmentto either the parity-mirror storage device 1602 or to the storage device150 storing the existing data segment and then verifies that a copy ofthe updated data segment is forwarded to the other device 1602, 150.

The apparatus 1800 includes a parity update module 1806 that calculatesone or more updated parity data segments for the one or moreparity-mirror storage devices of the existing stripe in response to astorage consolidation operation. The storage consolidation operation issimilar to the storage consolidation operation described above inrelation to the apparatus 1700 in FIG. 11. The storage consolidationoperation is conducted to recover at least storage space and/or data onone or more parity-mirror storage devices 1602 with the one or moreupdated parity data segments. By waiting to update the one or moreparity data segments, the update can be postponed until it is moreconvenient or until necessary to consolidate storage space.

In one embodiment, the updated parity data segment is calculated fromthe existing parity data segment, the updated data segment, and theexisting data segment. In one embodiment, the existing data segment ismaintained in place prior to reading the existing data segment forgeneration of the updated parity data segment. An advantage to thisembodiment, is the overhead associated from copying the existing datasegment to the parity-mirror storage device 1602 or other location wherethe updated parity data segment is generated can be postponed untilnecessary. A disadvantage of this embodiment is that if the storagedevice 150 that maintains the existing data segment fails, the existingdata segment must be recovered before the updated parity data segmentcan be generated.

In another embodiment, the existing data segment is copied to thedata-mirror storage device 1602 when the storage device 150 of the Nstorage devices 150 a-n where the existing data segment is storedreceives a copy of the updated data segment. The existing data segmentis then stored until the storage consolidation operation. In anotherembodiment, the existing data segment is copied to the data-mirrorstorage device 1602 in response to a storage consolidation operation onthe storage device 150 of the N storage devices 150 a-n where theexisting data segment is stored if the storage consolidation operationoccurs before the storage consolidation operation that triggerscalculation of the updated parity data segment. The latter embodiment isadvantageous because the existing data segment is not copied untilrequired by a storage consolidation operation on either the storagedevice 150 where the existing data segment is stored or on theparity-mirror storage device 1602.

In one embodiment, the updated parity data segment is calculated fromthe existing parity data segment, the updated data segment, and a deltadata segment, where the delta data segment is generated as a differencebetween the updated data segment and the existing data segment.Typically, generating a delta data segment is a partial solution orintermediate step in updating the parity data segment. Generating adelta data segment is advantageous because it may be highly compressibleand may be compressed before transmission.

In one embodiment, the delta data segment is stored on the storagedevice storing the existing data segment prior to reading the delta datasegment for generation of the updated parity data segment. In anotherembodiment, the delta data segment is copied to the data-mirror storagedevice 1602 when the storage device 150 where the existing data segmentis stored receives a copy of the updated data segment. In anotherembodiment, the delta data segment is copied to the data-mirror storagedevice 1602 in response to a storage consolidation operation on thestorage device 150 where the existing data segment is stored. As withcopying the existing data segment, the latter embodiment is advantageousbecause the delta data file is not moved until the earlier of a storageconsolidation operation on the storage device 150 storing the existingdata segment or another storage consolidation operation triggeringcalculation of the updated parity data segment.

In various embodiments, all of a portion of the actions of the modules1802, 1804, 1806, namely receiving an updated data segment, copying theupdated data segment, and calculating the updated parity data segment,occurs on a storage device 150 of the storage device set 1604, a client114, or a third-party RAID management device. In another embodiment, thestorage consolidation operation is conducted autonomously from theoperations of the update receiver module 1802 and the update copy module1804.

FIG. 13 is a schematic flow chart diagram illustrating an embodiment ofa method 1900 for managing data using progressive RAIDing in accordancewith the present invention. The method 1900 begins 1902 and the storagerequest receiver module 1702 receives 1904 a request to store data,where the data is data of a file or of an object. The striping module1704 calculates a stripe pattern for the data and writes 1906 the N datasegments to N storage devices 150. The stripe pattern includes one ormore stripes. Each stripe includes a set of N data segments where eachof the N data segments is written to a separate storage device 150within a set of storage devices 1604 assigned to the stripe.

The parity-mirror module 1706 writes 1908 a set of N data segments ofthe stripe to one or more parity-mirror storage devices 1602 within theset of storage devices 1604. The one or more parity-mirror storagedevices are in addition to the N storage devices 150 a-n. The paritygeneration module 1708 determines 1910 if there is a pending storageconsolidation operation. If the parity generation module 1708 determines1910 that there is no pending storage consolidation operation, themethod 1900 returns and again determines 1910 if there is a pendingstorage consolidation operation. In other embodiments, the storagerequest receiver module 1702, the striping module 1704, and theparity-mirror module 1706, continue to receive storage requests,calculate striping patterns, and storing data segments.

If the parity generation module 1708 determines 1910 that there is nopending storage consolidation operation, the parity generation module1708 calculates 1914 a parity data segment for the stripe. The paritydata segment is calculated from the N data segments stored on aparity-mirror storage device 1602. The parity generation module 1708stores 1912 the parity data segment on the parity-mirror storage device1602 and the method 1900 ends 1916. The storage consolidation operationis conducted autonomously from receiving 1904 a request to store N datasegments, writing 1906 the N data segments to the N storage devices, orwriting 1908 the N data segments to one or more parity-mirror storagedevices. The storage consolidation operation is conducted to recover atleast storage space or data on the parity-mirror storage device 1602.

FIG. 14 is a schematic flow chart diagram illustrating an embodiment ofa method 2000 for updating a data segment using progressive RAIDing inaccordance with the present invention. The method 2000 begins 2002 andthe update receiver module 1802 receives 2004 an updated data segmentwhere the updated data segment corresponds to an existing data segmentof an existing stripe. The update copy module 1804 copies 2006 theupdated data segment to the storage device 150 where the correspondingexisting data segment is stored and to the one or more parity-mirrorstorage devices 1602 corresponding to the existing stripe.

The parity update module 1806 determines 2008 if a storage consolidationoperation is pending. If the parity update module 1806 determines 2008that there is no pending storage consolidation operation, the parityupdate module 1806 waits for a storage consolidation operation. In oneembodiment, the method 2000 returns and receives 2004 other updated datasegments and copies 2006 the updated data segments. If the parity updatemodule 1806 determines 2008 that there is no pending storageconsolidation operation, the parity update module 1806 calculates 2010one or more updated parity data segments for the one or moreparity-mirror storage devices of the existing stripe and the method 2000ends 2012.

Front-End Distributed Raid

Traditional RAID systems are configured with a RAID controller thatfunctions to receive data, calculate striping patterns for the data,divide the data into data segments, calculate a parity stripe, store thedata on storage devices, update the data segments, etc. While some RAIDcontrollers allow some functions to be distributed, the storage devicesmanaged by the RAID controller do not communicate with clients 114directly for storing data striped in a RAID. Instead storage requestsand data for RAIDing pass through the storage controller.

Requiring the RAID controller to touch all of the data to be stored in aRAID is inefficient because it creates a dataflow bottleneck. This isespecially true during a read-modify-write process where bandwidth andperformance of all of the drives in the RAID group is consumed whileonly a subset is actually updated. In addition, the a region of thestorage device designated for data managed by the RAID controller istypically dedicated to the RAID group and cannot be accessedindependently. Access to a storage device 150 by a client must typicallybe accomplished by partitioning the storage device 150. Wherepartitioning is used, partitions accessible for general storage are notused for RAID and partitions allocated to the RAID group are notaccessible for general data storage. Schemes that oversubscribepartitions in order to globally optimize utilization are complex andmore difficult to manage. In addition, storage space allocated for oneRAID group cannot be accessed by more than one RAID controller unlessone is designated as master and other RAID controllers act as slavesunless the master RAID controller is inactive, non-functional, etc.

Typical RAID controllers also generate parity data segments outside ofthe storage devices 150 of the RAID group. This can be inefficientbecause parity data segments are typically generated and then are sentto a storage device 150 for storage, which requires computing capacityof the RAID controller. Tracking parity data segment location andupdates must also be done at the RAID controller instead of autonomouslyat a storage device 150.

Where it is necessary to ensure that the data remains available if theseparate RAID controller is offline, RAID controllers are typicallycross connected to the drives and to each other, and/or mirrored ascomplete sets, making data availability expensive and difficult tomanage, and dramatically reducing the reliability of the storagesubsystem.

What is needed is a system, apparatus, and method for front-enddistributed RAID that allows RAIDing on a per data segment, per object,per file, or similar basis and that eliminates the need for RAIDcontrollers and RAID controller couplets situated between the client andthe storage devices. In such a system, apparatus, and method, a RAIDgroup can be created for one data segment, object, or file and managedwithin one group of storage devices by one RAID controller while asecond RAID group may be created for another data segment, object, orfile that encompasses some of the same storage devices of the first RAIDgroup. The RAID control functions may be distributed among clients 114,a third-party RAID management device, or among storage devices 150. Thefront-end distributed RAID system, apparatus, and method may also sendcommands to storage devices 150 of a RAID group and may allow thestorage devices 150 to directly access and copy data through directmemory access (“DMA”), or remote DMA (“RDMA”).

FIG. 10 is a schematic block diagram illustrating one embodiment of asystem 1600 that may be accessed for a front-end distributed RAID inaccordance with the present inventions. The descriptions above for thecomponents depicted in FIG. 16 related to progressive RAID are alsoapplicable to front-end distributed RAID. With respect to front-enddistributed RAID, the storage device set 1604 forms a RAID group andincludes storage devices 150 that are autonomous and are capable ofindependently receiving and servicing storage requests from a client 114over a network 116 or one or more redundant networks 116.

Among the storage devices 150 within the storage device set 1604, one ormore are designated as parity-mirror storage devices 1602 for a stripe.Typically, the one or more parity-mirror storage devices 1602 functionsubstantially similar to the other storage devices 150. In typicalconfigurations where the designated parity-mirror storage devices 1602alternate among the storage devices 150 of the storage device set 1604,the parity-mirror storage devices 1602 have essentially the samecharacteristics as the other storage devices 150 because they must alsooperate as non-parity-mirror storage devices. The similarcharacteristics are with respect to operation within a RAID group andautonomous operation for independent client 114 communication asdescribed above. In various embodiments, the storage devices 150 of thestorage device set 1604 may differ in other aspects not related tofunctioning within the described RAID environment.

The storage devices 150 of the storage device set 1604 may be standalone, grouped within one or more servers 112, may each reside in aserver 112, may be accessed through one or more servers 112, etc. One ormore clients 114 may reside in servers 112 that include one or morestorage devices 150, may reside in separate servers 112, may reside incomputers, workstations, laptops, etc. that access the storage devices150 through one or more computer networks 116, or the like.

In one embodiment, the network 116 includes a system bus and one or moreof the storage devices 150, 1602 of the storage device set 1604communicate using the system bus. For example, the system bus may be aPCI-e bus, a Serial Advanced Technology Attachment (“serial ATA”) bus,parallel ATA, or the like. In another embodiment, the system bus is anexternal bus such as small computer system interface (“SCSI”), FireWire,Fiber Channel, USB, PCIe-AS, Infiniband, or the like. One of skill inthe art will appreciate other system 1600 configurations with storagedevices 150 that are autonomous and are capable of independentlyreceiving and servicing storage requests from a client 114 over one ormore networks 116.

FIG. 15 is a schematic block diagram illustrating one embodiment of anapparatus 2100 for front-end distributed RAID in accordance with thepresent invention. The apparatus 2100, in various embodiments, includesa storage request receiver module 2102, a striping association module2104, a parity-mirror association module 2106, a storage requesttransmitter module 2108, a front-end parity generation module 2110, aparity alternation module 2112, a data segment recovery module 2114, adata rebuild module 2116, a parity rebuild module 2118, and apeer-to-peer communication module 2120, which are described below. Invarious embodiments, the apparatus 2100 may be included in a storagedevice 150 such as a solid-state storage device 102, a storage devicecontroller 152 such as a solid-state storage controller 104, a server112, a third-party RAID management device, etc. or may be distributedamong more than one component.

The apparatus 2100 includes a storage request receiver module 2102 thatreceives a storage request to store data in a storage device set 1604.The data may be a portion of a file or an object or may be an entirefile or object. A file may include a block of arbitrary information, orresource for storing information, which is available to a computerprogram. A file may include any data structure accessed by a processor.A file may include a database, a string of text, computer code, etc. Anobject is typically a data structure for object-oriented programming andmay include a structure with or without data. In one embodiment, anobject is a subset of a file. In another embodiment, an object isindependent of a file. In any case, an object and a file are definedherein to include the entire set of data, data structures, computercode, and other information that may be stored on a storage device.

The storage device set 1604 includes autonomous storage devices 150forming a RAID group that independently receive storage requests from aclient 114 over one or more networks 116. One or more of the autonomousstorage devices 150 within the storage device set 1604 are designated asparity-mirror storage devices 1602 for a stripe. Other storage requestsfrom another client 114 may be stored on a second storage device setwhere the second storage device set may include one or more of the samestorage devices 150 (and parity-mirror storage devices 1602) as thefirst storage device set 1604. The storage devices 150 common to bothstorage device sets 1604 may have overlapping assigned storage spacewithin the common storage devices 150.

The apparatus 2100 includes a striping association module 2104 thatcalculates a stripe pattern for the data. The stripe pattern includesone or more stripes. Each stripe includes of a set of N data segments.The N data segments of a stripe may also include one or more empty datasegments. The striping association module 2104 associates each of the Ndata segments with one of N storage devices 150 a-n in the storagedevice set 1604 assigned to the stripe. In one embodiment, the stripingassociation module 2104 associates a data segment with a storage device150 with a storage request to be sent the storage device 150 thatdirects the storage device 150 to get data corresponding to the datasegment from the client 114 sending the storage request.

In another embodiment, the storage request is substantially free of dataof the data segments. Substantially free of data means that the storagerequest generally does not include the data that is the subject of thestorage request but may include characters, character strings, etc. thatmay be part of the data. For example, if the data comprises a series ofrepeated, identical characters, such as a series of zeros, the storagerequest may include an indication that the data includes a series ofzeros without including all of the zeros contained in the data. One ofskill in the art will recognize other ways to send a storage requestwithout sending the bulk of the data while still allowing a small amountor single instance of certain characters or character strings in thestorage request. The storage request includes commands that allow the Nstorage devices 150 a-n to retrieve the data using a DMA or RDMAoperation or the like.

In another embodiment, the striping association module 2104 associates adata segment with a storage device 150 by identifying in a storagerequest to be sent to the storage device 150 the data of the datasegment. Identifying the data of the data segment may include a datasegment identifier, a data segment location or address, a data segmentlength, or other information that will allow the storage device 150 torecognize which data comprises the data segment.

In one embodiment, the striping association module 2104 associates adata segment with a storage device 150 in a storage request so that aclient 114 can send data comprising the data segments in a broadcast sothat each storage device 150 is able to store associated data segmentsand discard data corresponding to data segments not assigned to thestorage device 150. In another embodiment, the striping associationmodule 2104 associates a data segment with a storage device 150 in astorage request, possibly by addressing each data segment, so that aclient 114 can send data comprising the data segments in a multicast sothat each storage device 150 is able to store associated data segmentsand discard data corresponding to data segments not assigned to thestorage device 150. One of skill in the art will recognize other waysfor the striping module 2104 to associate a data segment with a storagedevice 150 to broadcast, multicast, unicast, anycast, etc. one or moredata segments to the one or more storage devices.

In a related embodiment, the striping association module 2104 associatesa data segment with a storage device 150 in a storage request so that aclient 114 can broadcast, multicast, unicast, etc. the storage requestand each storage device 150 is able to receive a portion of the storagerequest from the client 114 pertaining to the data segment associatedwith the storage device 150 and can discard portions the storage requestthat do not pertain to the one or more data segments not associated withthe storage device 150.

In another embodiment, the storage request received by the storagerequest receiver module 2102 includes the data that is the subject ofthe storage request and the striping association module 2104 associatesa data segment with a storage device 150 by preparing a storage requestfor the storage device 150 that includes data segment. The stripingassociation module 2104 may operate within a client 114, a third-partyRAID management device, a storage device 150, 1602, etc.

The apparatus 2100 includes a parity-mirror association module 2106 thatassociates a set of the N data segments with one or more parity-mirrorstorage devices 1602 in the storage device set 1604. The one or moreparity-mirror storage devices 1602 are in addition to the N storagedevices 150 a-n. In one embodiment, the parity-mirror association module2106 associates a set of N data segments to each parity-mirror storagedevice 1602 so each parity-mirror storage device 1602 can receive andstore the N data segments of a stripe for generating a parity datasegment. In another embodiment, the parity-mirror association module2106 associates a data segment of the stripe with each parity-mirrorstorage device 1602 so that the parity-mirror storage devices 1602 a-mact as a mirror to the N data segments stored on the N storage devices150 a-n.

In various embodiments, the parity-mirror association module 2106associates the set of N data segments with the one or more parity-mirrorstorage devices 1602 using a single storage request, multiple storagerequests, or other association technique described above in relation tothe striping association module 2104, such as storage requests settingthe parity-mirror storage device 1602 up for DMA, RDMA, broadcasts,multicasts, or including the N data segments in the storage requests.The parity-mirror association module 2106 may operate within a client114, a third-party RAID management device, a storage device 150, 1602,etc.

The apparatus 2100 includes a storage request transmitter module 2108that transmits one or more storage requests to each storage device 150,1602 in the storage device set 1604, each storage request sufficient tostore onto the storage device 150, 1602 the one or more data segmentsassociated with the storage device 150, 1602 receiving the storagerequest. In one embodiment, each storage request does not include datathat is the subject of the storage request. In a further embodiment,each storage request enables the N storage devices 150 and parity-mirrorstorage devices 1602 of the storage device set 1604 to download data ofan associated data segment using DMA or RDMA. In another embodiment, astorage request contains sufficient information to pick out relevantstorage requests or relevant data for the associated data segments froma broadcast from the client 114. In another embodiment, a storagerequest includes the data of an associated data segment.

In one embodiment, each storage request identifies the storage devices150, 1602 that are part of the storage device set 1604 of a stripe. Byincluding an identification of the storage devices 150, 1604 of thestorage device set 1604, in the event of a failure of a storage device150 acting as master, another storage device 150 may take over as masterfor managing the RAIDed data. In another embodiment, the identificationof the storage device set 1604 enables the autonomous storage devices150, 1602 to recover data when a storage device is off-line, and rebuilddata when a replacement storage device is added within the storagedevice set 1604 independent of the client. In another embodiment, theidentification of the storage devices 150, 1602 of the storage deviceset 1604 represents a multi-cast group for transmission of data segmentsor storage requests. The identification may be stored along withmetadata for the object or file stored on the storage devices 150, 1602within the storage device set 1604.

In one embodiment, when the parity-mirror association module 2106associates a set of N data segments with each of the one or moreparity-mirror storage devices 1602, the apparatus 2100 includes afront-end parity generation module 2110 that calculates, independent ofa client 114, a parity data segment for the stripe and stores the paritydata segment on the parity-mirror storage device 1602. The parity datasegment is calculated from the set of N data segments provided to theparity-mirror storage device 1602. Where more than one parity-mirrorstorage device 1602 is included in the storage device set 1604, thefront-end parity generation module 2110 typically generates the variousparity data segments so that two or more storage devices 150, 1602 inthe storage device set 1604 may fail and the parity data segmentinformation allows for recovery of unavailable data segments or paritydata segments.

In another embodiment, the front-end parity generation module 2110calculates the parity data segment when operating within a storagedevice 150 of the storage device set 1604 and/or a third party RAIDmanagement device. For example, a server 112 separate from the client114 transmitting the storage request may calculate the parity datasegment. In another embodiment, the front-end parity generation module2110 operates within a parity-mirror storage device to calculate theparity data segment. For example, a storage controller 152 in theparity-mirror storage device 1602 may act as a master storage controllerfor the RAID group formed by the storage device set 1604.

In another embodiment, the front-end parity generation module 2110calculates the parity data segment and then transmits the calculatedparity data segment to one or more additional parity-mirror storagedevices 1604 in a second set of storage devices forming a mirror. Thisembodiment is advantageous because overhead associated with calculatinga parity data segment is expended once rather than for each storagedevice set 1604 with the additional benefit of reducing the data trafficon the network 116.

The apparatus 2100 may also include, in one embodiment, a data segmentrecovery module 2112 that recovers a data segment stored on a storagedevice 150 of the storage device set 1604 if the storage device 150 isunavailable and a request is received to read either the unavailabledata segment or data that includes the unavailable data segment. Thedata segment is recovered using the data segments on available storagedevices 150 of the storage device set 1604, a combination of the paritydata segments and the data segments on available storage devices 150,1602 of the storage device set 1604, or from a mirror storage devicecontaining a copy of the data segment. Typically, the mirror storagedevice is one storage device 150 of a mirror storage device set thatstores a copy of the N data segments. The data segment recovery module2112 may operate and recover the unavailable data segment from a storagedevice 150, a parity-mirror storage device 1602, a third-party RAIDmanagement device, a mirror storage device, etc.

In another embodiment, the apparatus 2100 includes a data rebuild module2114 that stores a recovered data segment on a replacement storagedevice 150 in a rebuild operation. For example, if a storage device 150becomes unavailable due to failure, loss of synchronization, etc. thedata rebuild module 2114 may rebuild a storage device 150 to replace theunavailable storage device 150. In one embodiment, the rebuilt storagedevice 150 is the original storage device 150 that has been madeavailable.

The recovered data segment matches an unavailable data segment stored onthe unavailable storage device 150 of the storage device set 1604. Therebuild operation typically restores one or more of data segments andparity data segments onto the replacement storage device 150 to matchdata segments and parity data segments stored previously on theunavailable storage device 150.

In one embodiment, the recovered data segment is recovered for therebuild operation using the available data segments on available storagedevices 150 of the storage device set 1602. In another embodiment, therecovered data segment is recovered for the rebuild operation using acombination of a parity data segment from one or more of theparity-mirror storage devices 1602 and the available data segments onavailable storage devices 150 of the storage device set 1604. In anotherembodiment, the recovered data segment is recovered for the rebuildoperation using a matching data segment read from a parity-mirrorstorage device 1602. In yet another embodiment, the recovered datasegment is recovered for the rebuild operation using a matching datasegment from a mirror storage device. The data rebuild module 2114 couldoperate and store a received data segment from a client 114, a thirdparty RAID management device, a storage device 150, 1602, a mirrorstorage device, etc.

The apparatus 2100 includes, in another embodiment, a parity rebuildmodule 2116 that rebuilds the recovered parity data segment on areplacement storage device 1602 in a rebuild operation. A rebuildoperation is substantially similar to the rebuild operation described inrelation to the data rebuild module 2114. The parity rebuild module 2116operates similar to the data rebuild module 2114 except the parityrebuild module 2116 rebuilds a recovered parity data segment. Therecovered parity data segment matches an unavailable parity data segmentstored on an unavailable parity-mirror storage device 1602 assigned tothe stripe.

The parity data segment is recovered, in various embodiments, by copyingthe parity data segment stored on a parity-mirror storage device 1602 ina mirrored storage device set, copying the parity data segment from aparity-mirror storage device 1602 in the storage device set 1604 (ifidentical to the unavailable parity data segment), generating the paritydata segment using one or more of the N data segments and parity datasegments stored on the available storage devices 150, 1602 of thestorage device set 1604 and a mirror storage device containing a copy ofa data segment, etc. The data rebuild module 2116 may operates andstores a recovered data segment while residing on a client 114, a thirdparty RAID management device, a storage device 150, a mirror storagedevice, etc.

Advantageously, the apparatus 2100 is not limited to storing data in thestorage devices 150, 1602 to partitions dedicated to the front-end,distributed RAID operation described herein. Instead, an autonomousstorage device (e.g. 150 a) may independently receive storage requestsfrom a client 114 to store RAIDed or un-RAIDed data in one or moreregions of the storage device 150 a that is also available for storingdata by the striping association module 2104, the parity-mirrorassociation module 2106, and the front-end parity generation module2110.

In one embodiment, one or more storage requests received by the storagerequest receiver module 2102 or transmitted by the storage requesttransmitter module 2108, identify the storage devices 150 that comprisethe storage device set 1604 of the stripe. Advantageously, identifyingthe storage device 150 of the storage device set 1604 in the storagerequests facilitates a backup RAID controller to operate if a mastercontroller is non-functional. For example, if the storage devices 150 ofthe storage device set 1604 are identified in storage requests and themaster controller is in a parity-mirror storage device 1602 and isunavailable, another parity-mirror storage device 1602 or another of theN storage devices 150 a-n may become the master controller.

In one embodiment, the apparatus 2100 includes a parity alternationmodule 2118 that alternates, for each stripe, which storage devices 150in the storage device set 1604 are designated as the parity-mirrorstorage devices 1602 for the stripe. The benefits of the parityalternation module 2118 are described above. In another embodiment, thestorage devices 150 of the storage device set 1604 form a group of peersand the apparatus 2100 includes a peer-to-peer communication module 2120that transmits and receives storage requests within the storage devices150, 1602 of the storage device set 1604. The peer-to-peer communicationmodule 2120 may also transmit and receive storage requests with peerdevices outside the storage device set 1604.

In a preferred embodiment, the storage request is an object request tostore an object by striping data of the object across the storagedevices 150, 1602 of the storage device set 1604 using the modules2102-2120 of the apparatus 2100. In another embodiment, one or more ofthe autonomous storage devices 150, 1602 of the storage device set 1604are allocated within a first RAID group for at least a portion of afirst object or file and allocated within a second RAID group for atleast a portion of a second object or file. For example, one storagedevice 150 a may be a master RAID controller for the storage device set1604 for one or more stripes and a second storage device 150 b may be amaster RAID controller for a RAID group that includes some or all of thestorage devices 150 of the storage device set. Advantageously, theapparatus 2100 allows flexibility in grouping storage devices 150, 1602to form RAID groups for various clients 114.

FIG. 16 is a schematic flow chart diagram illustrating an embodiment ofa method 2200 for front-end distributed RAID in accordance with thepresent invention. The method 2200 begins 2202 and the storage requestreceiver module 2102 receives 2204 a storage request to store data in Nstorage devices 150 a-n of the storage device set 1604. The stripingassociation module 2104 calculates 2206 a stripe pattern for the dataand associates 2208 the N data segments each with one of the N storagedevices 150 a-n.

The parity-mirror association module 2106 associates 2210 a set of Ndata segments with one or more parity-mirror storage devices 1602. Thestorage request transmitter module 2108 transmits 2212 one or morestorage requests to each storage device 150, 1602 in the storage deviceset 1604. Each storage request is sufficient to store onto the storagedevice 150 the one or more data segments associated with the storagedevice 150 receiving the storage request. The data segments of the dataare then transferred to the storage devices 150, 1602 of the storagedevice set 1604 using DMA, RDMA, broadcast, multicast, etc. as directedby the storage requests. Optionally, the front-end parity generationmodule 2110 calculates 2214 a parity data segment for the stripe and themethod 2200 ends 2216.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

1. A method for providing high performance, distributed storage,comprising: receiving a storage request at an autonomous storage devicein a storage device set comprising a plurality of storage devices, thestorage request identifying one or more data segments of a data stripepattern assigned to the storage device by a front-end controller; andthe autonomous storage device communicating the identified data segmentswith a storage client independently of the front-end controller.
 2. Themethod of claim 1, wherein the autonomous storage device communicatesdata directly with the storage client by way of one of a bus, directmemory access, remote direct memory access, network broadcast, networkmulticast, network anycast, and network unicast.
 3. The method of claim1, wherein the identified data segments comprise parity data of the datastripe pattern.
 4. The method of claim 1, wherein the storage device setcomprises a redundant array of independent disks (“RAID”) group, themethod further comprising: maintaining indicators of the storage devicesin the RAID group on the autonomous storage device; and reconstructingthe RAID group using the indicators of the autonomous storage device. 5.The method of claim 1, further comprising: receiving a second, differentstorage request at the autonomous storage device pertaining to a clientstorage request that is independent of the storage device set; andstoring data of the second storage request on the autonomous storagedevice with the identified data segments.
 6. The method of claim 1,further comprising: receiving a second storage request at the autonomousstorage device referencing a second storage device set comprising adifferent plurality of storage devices, wherein the second storagerequest identifies one or more second data segments of a second datastripe pattern assigned to the storage device by the front-endcontroller; and the autonomous storage device transferring the secondidentified data segments with a storage client independently of thefront-end controller.
 7. The method of claim 1, further comprising:storing the identified data segments sequentially by order of processingon a solid-state storage media of the autonomous storage device; markingstorage divisions of the solid-state storage media with respectivesequence indicators; and reconstructing an index of data stored on theautonomous storage device based on the sequence indicators.
 8. Themethod of claim 1, wherein the autonomous storage device comprises aplurality of solid-state storage elements, each solid-state storageelement comprising a plurality of storage divisions, the method furthercomprising storing the identified data segments on two or more of thesolid-state storage elements in parallel.
 9. The method of claim 1,further comprising: including data of the identified data segments in anerror-correcting code (ECC) block that comprises an error-correctingcode configured to correct an error in the ECC block; and storing afirst portion of the ECC block on a first one of a plurality ofsolid-state storage elements; and storing a second portion of the ECCblock on a second one of the plurality of solid-state storage elements.10. An autonomous storage device in storage device set comprising aplurality of storage devices, the autonomous storage device comprising:a storage request receiver configured to receive a storage requestidentifying one or more data segments of a data stripe pattern assignedto the autonomous storage device by a front-end controller; and astorage controller configured to communicate the identified datasegments with a storage client independent of the front-end controller.11. The autonomous storage device of claim 10, wherein the autonomousstorage device communicates data directly with the storage client by wayof one of a bus, direct memory access, remote direct memory access,network broadcast, network multicast, network anycast, and networkunicast.
 12. The autonomous storage device of claim 10, wherein theidentified data segments comprise parity data of the data stripepattern.
 13. The autonomous storage device of claim 10, wherein thestorage device set comprises a redundant array of independent disks(“RAID”) group, and wherein the autonomous storage device maintainsindicators of storage devices in the RAID group and is configured toreconstruct the RAID group using the indicators.
 14. The autonomousstorage device of claim 10, further comprising a sequential storagemodule to store the identified data segments on a non-volatile storagemedia with data that does not pertain to the storage device set.
 15. Theautonomous storage device of claim 10, further comprising a sequentialstorage module to mark storage divisions of a non-volatile storage mediaof the autonomous storage device with respective sequence indicators toindicate an order of storage operations performed on the autonomousstorage device.
 16. The autonomous storage device of claim 10, furthercomprising: a plurality of solid-state storage elements, eachsolid-state storage element comprising a plurality of storage divisions;and a sequential storage module configured to store the identified datasegments on two or more of the solid-state storage elements in parallel.17. The autonomous storage device of claim 10, further comprising: anECC generator to generate an error-correcting code (ECC) blockcomprising data of the identified data segments and an error-correctingcode; and a sequential storage module configured to store a firstportion of the ECC block on a first one of a plurality of solid-statestorage elements and a second portion of the ECC block on a second oneof the plurality of solid-state storage elements.
 18. A storage system,comprising: a front-end controller; and a plurality of autonomousstorage devices in a storage device set, each autonomous storage devicecomprising: a storage request receiver to receive a storage requestidentifying one or more data segments of a data stripe pattern that areassigned to the autonomous storage device by a front-end controller, anda storage controller configured to communicate the identified datasegments assigned to the respective autonomous storage device with astorage client independent of the front-end controller.
 19. The storagesystem of claim 18, wherein the front-end controller operates on one ofthe autonomous storage devices in the storage device set.
 20. Thestorage system of claim 18, wherein each of the autonomous storagedevices in the storage device set communicate data directly with thestorage client in response to a storage request from the front-endcontroller by way of one of a bus, direct memory access, remote directmemory access, network broadcast, network multicast, network anycast,and network unicast.